Introduction
Speech-to-speech (S2S) translation systems [1] combine various technologies to help to translate speech across languages.  However, most S2S sys- tems ignore paralinguistic information such as em- phasis.  This paper attempts to solve the problem by proposing two new components: word-level em- phasis estimation [2] using linear regression hidden semi-Markov models (LR-HSMM) [3], and empha- sis translation that translates the word-level empha- sis to a target language with conditional random fields (CRFs) [4].  The result shows that our sys- tem can accurately translate emphasis with 91.6% F -measure according to objective evaluation.  A lis- tening test with human subjects further showed that they could identify emphasized words with 87.8% F- measure.  This paper is a shortened version of our work presented in [5]. 2
Word-level Emphasis Translation
Our proposed emphasized speech translation model consists of a conventional S2S system, an em- phasis estimation model, and an emphasis transla- tion model as illustrated in Fig.  1.  As described in detail in our previous work [2], the emphasis es- timation component estimates an emphasis-level se- quence Â for every utterance using LR-HSMM. Each word in the utterance has its own emphasis level rep- resenting how emphasized the word is. Adopting the result from [2], our next step is emphasis translation, or to take the estimated em- phasis sequence in the source languages î(f), and convert them to an emphasis sequence in the tar- get language î 'c' using CRFs.  In order to train the CRF model, we create training data consist- ing of source and target words w(f) and w(c), and the corresponding estimated emphasis values.  As λ (f) is a sequence of continuous values, and CRFs requires discrete state sequences, we first quan- and  λ c  i into buckets, giving us a dis- tize  λ(f) and  Î (c)' crete sequence  {{f}' We then create CRFs training data that consists of N samples D = [(x 1 , λ (ε)′ ), · , · , (x n, λ (ε)′ ), ·· , (x n, λ (ε)′ )], where x n is a feature vector consisting of: and its con- ● source word-level emphasis λ(f), text, ● source word w ( f ) , and word context, ● source part of speech (PoS), and PoS context, ● target word w n ', and word context, ● target PoS, and PoS context, where context means the information of one suc- ceeding and one preceding words. To decide which source features correspond to a target word w n ' , we use one-to-one word alignments between w f ) and w ( ε ). The likelihood of CRFs is given by
where θ k is model parameters, f is feature func- tion, and K is number of feature function. Different combination of input features described above will be used depends on individual feature function fk.
3
Experiments
Experiments were performed using a bilingual English-Japanese emphasis corpus [6].  The corpus consists of 1015 parallel utterances of English and Japanese.  We use the data from 2 speakers, a na- tive English speaker, and a Japanese native speaker as the training and testing data.  After filtering out F -measure for different combinations of Table  1 input features.  e_en and e_ja denote word-level em- phasis, w_en and w_ja denote word information, t_en
long sentences over 10 words, we obtained 966 ut- terances, which we divided into 916 sentences with 1,186 emphasized words for training and 50 sen- tences with 62 emphasized words for testing. The word-level emphasis is quantized to the closest of {0, 0.3, 0.6, 0.9}. To measure the accuracy, we calculate emphasis F-measure, the harmonic mean of the precision and recall with which the system detects emphasis.
Emphasis translation evaluation
3.1
In our first experiment, we evaluate the ability of the proposed method to reproduce emphasis in the target language. In addition, we also evaluate the effect of the combination of input features described in Section 2 to find out which features give the high- est F-measure. The result is shown in Table 1. We can see that the model that use the features including the source language information (2nd-6th row) is better than the model use only target infor- mation (1st row).  This demonstrates that our model effectively translates emphasis from the source, as opposed to simply predicting based on the target. Looking at the second and third rows, we can see that emphasis contexts in the source language does not help word-level emphasis translation, indicat- ing that word-level emphasis in the target language depends mainly on emphasis of the corresponding source word. By adding more linguistic information such as word and PoS, the best system is achieved with 91.6% F-measure. In addition, we carried out a listening evaluation with 6 native Japanese speakers.  We asked to 6 native Japanese speakers to listen to the emphasis translated utterances, and select the words that they think are emphasized in 150 randomized testing ut- terances from the following 3 systems:  Baseline— no emphasis translation, CRF-based-with CRF- based emphasis translation, and Natural–natural spoken speech by a Japanese speaker. The result in Fig. 2 shows that the CRF-based system achieved 87.8% F-measure that outperforms the Baseline system with 77.0% F -measure.
4
Conclusion
In this paper, we present a new speech translation architecture that is able to translate emphasis across languages. Experiments showed that our proposed approach can accurately convey emphasis in speech translation models. Future work will improve the emphasized speech synthesis quality, and the em- phasis translation model.
Acknowledgements
Part of this research was supported by JSPS KAKENHI Grant Number 24240032 and 26870371, and the Commissioned Research of National Insti- tute of Information and Communications Technol- ogy (NICT), Japan.