3-1-8
Word-level Emphasis Transfer in Speech-to-speech Translation ∗☆ Quoc Truong Do, Shinnosuke Takamichi, Sakriani Sakti, Graham Neubig(NAIST),
Tomoki Toda (NAIST/Nagoya University), Satoshi Nakamura (NAIST)

1Introduction(e)Speech-to-speech (S2S) translation systems [1]combine various technologies to help to translatespeech across languages. However, most S2S systems ignore paralinguistic information such as emphasis. This paper attempts to solve the problem
by proposing two new components: word-level emphasis estimation [2] using linear regression hidden
semi-Markov models (LR-HSMM) [3], and emphasis translation that translates the word-level emphasis to a target language with conditional randomﬁelds (CRFs) [4]. The result shows that our system can accurately translate emphasis with 91.6%
F -measure according to objective evaluation. A listening test with human subjects further showed thatthey could identify emphasized words with 87.8% F measure. This paper is a shortened version of ourwork presented in [5].

2Word-level Emphasis Translation

Our proposed emphasized speech translation
model consists of a conventional S2S system, an emphasis estimation model, and an emphasis translation model as illustrated in Fig. 1. As described
ASRItEmphasis est.

Itis

hot

0.1

0.2

0.9

Emphasis trans.

Fig. 1

暑いです

0.8

0.1

is

hot

tize λ̂(f )(e)and λ̂

(e)′(f )• source word-level emphasis λj , and its context,(f )• source word wj , and word context,• source part of speech (PoS), and PoS context,(e)• target word wn , and word context,• target PoS, and PoS context,
where context means the information of one succeeding and one preceding words.
To decide which source features correspond to a(e)target word wn , we use one-to-one word alignments(f )(e)between wj and wn . The likelihood of CRFs isgiven by}
{KN
∑
∏
′(e)′
(k)
θk fk (λn−1 , λ(e)exp
n , xn )です

n=1N∑ ∏λ̃TTS(e)′ n=1exp

k=1{K∑},′(e)′(k)θk fk (λ̃n−1 , λ̃(e)n , xn )k=1(1)
where θk is model parameters, f is feature function, and K is number of feature function. Diﬀerent
combination of input features described above will
be used depends on individual feature function fk .

Emphasis speech-to-speech translation(f )into buckets, giving us a dis-(f )′crete sequence λ̂and λ̂ . We then create
CRFs training data that consists of N samples D =(e)′(e)′(e)′[(x1 , λ1 ), · · · , (xn , λn ), · · · , (xN , λN )], where xn
is a feature vector consisting of:

P (λ(e) |x) =in detail in our previous work [2], the emphasis estimation component estimates an emphasis-level sequence λ̂ for every utterance using LR-HSMM. Eachword in the utterance has its own emphasis level representing how emphasized the word is.
Adopting the result from [2], our next step isemphasis translation, or to take the estimated emphasis sequence in the source languages λ̂ , andconvert them to an emphasis sequence in the tar∗

(f )

λ̂
is a sequence of continuous values, and CRFs
requires discrete state sequences, we ﬁrst quan-′MT
暑いget language λ̂using CRFs. In order to train
the CRF model, we create training data consisting of source and target words w(f ) and w(e) , andthe corresponding estimated emphasis values. As

3Experiments

Experiments were performed using a bilingualEnglish-Japanese emphasis corpus [6]. The corpus
consists of 1015 parallel utterances of English andJapanese. We use the data from 2 speakers, a native English speaker, and a Japanese native speakeras the training and testing data. After ﬁltering out

音声翻訳システムにおける単語強調の伝達

日本音響学会講演論文集- 87 -2016年3月

w ja, t ja, t ja c, e ja
e en, e ja
e en, e ja, e en ce en, e ja, w en, w ja
e en, e ja, w en, w ja, t en, t ja
e en, e ja, w en, w ja, t en, t ja, t ja cbased emphasis translation, and Natural–natural
spoken speech by a Japanese speaker. The result in

81.6
82.8
82.8
84.8
90.0
91.6

90

87.8

85
80

77.0

75
70

long sentences over 10 words, we obtained 966 utterances, which we divided into 916 sentences with
1,186 emphasized words for training and 50 sentences with 62 emphasized words for testing. The
word-level emphasis is quantized to the closest of {0,0.3, 0.6, 0.9}.
To measure the accuracy, we calculate emphasis
F -measure, the harmonic mean of the precision andrecall with which the system detects emphasis.
Emphasis translation evaluation

In our ﬁrst experiment, we evaluate the ability ofthe proposed method to reproduce emphasis in thetarget language. In addition, we also evaluate theeﬀect of the combination of input features describedin Section 2 to ﬁnd out which features give the highest F -measure. The result is shown in Table 1.
We can see that the model that use the features
including the source language information (2nd-6throw) is better than the model use only target information (1st row). This demonstrates that our modeleﬀectively translates emphasis from the source, asopposed to simply predicting based on the target.
Looking at the second and third rows, we can seethat emphasis contexts in the source language does
not help word-level emphasis translation, indicating that word-level emphasis in the target language
depends mainly on emphasis of the corresponding
source word. By adding more linguistic informationsuch as word and PoS, the best system is achieved
with 91.6% F -measure.
In addition, we carried out a listening evaluation
with 6 native Japanese speakers. We asked to 6
native Japanese speakers to listen to the emphasis
translated utterances, and select the words that they
think are emphasized in 150 randomized testing utterances from the following 3 systems: Baseline–
no emphasis translation, CRF-based–with CRF-

日本音響学会講演論文集

98.0

95Fig. 2

3.1

95 % confidence interval

100F-measure

Table 1 F -measure for diﬀerent combinations ofinput features. e en and e ja denote word-level emphasis, w en and w ja denote word information, t en
and t ja denote PoS tag of English, and Japanese,respectively, and * c denotes context information.
Feature typeF -measure

Baseline

CRF-based

NaturalPrediction F -measure for manual evaluation

Fig. 2 shows that the CRF-based system achieved
87.8% F -measure that outperforms the Baseline
system with 77.0% F -measure.

4Conclusion

In this paper, we present a new speech translationarchitecture that is able to translate emphasis acrosslanguages. Experiments showed that our proposed
approach can accurately convey emphasis in speechtranslation models. Future work will improve theemphasized speech synthesis quality, and the emphasis translation model.
Acknowledgements
Part of this research was supported by JSPS
KAKENHI Grant Number 24240032 and 26870371,
and the Commissioned Research of National Institute of Information and Communications Technology (NICT), Japan.

References[1] S. Nakamura. Overcoming the language barrier withspeech translation technology. Science & Technology
Trends - Quarterly Review No.31, April 2009.
[2] D. Q. Truong, S. Takamichi, S. Sakti, G. Neubig, T. Toda,and S. Nakamura. Study on word-level emphasis acrossEnglish and Japanese. In Proceedings of ASJ, Autumn
meeting, 2015.
[3] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi. A
style control technique for HMM-based expressive speechsynthesis. IEICE, E90-D(9):1406–1413, September 2007.
[4] J.D. Laﬀerty, A. McCallum, and F.C.N. Pereira. Conditional random ﬁelds: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML,pages 282–289, 2001.
[5] S. Sakti G. Neubig T. Toda S. Nakamura Q. T. Do,S. Takamichi. Preserving word-level emphasis in speechto-speech translation using linear regression HSMMs. In
INTERSPEECH, 2015.
[6] D. Q. Truong, S. Sakti, G. Neubig, T. Toda, and S. Nakamura. Collection and analysis of a Japanese-English emphasized speech corpus. In Proceedings of Oriental COCOSDA, September 2014.

- 88 -2016年3月