End to end Model for Cross-Lingual Transformation of
Paralinguistic Information
Takatomo Kano, Shinnosuke Takamichi, Sakriani Sakti, Graham Neubig,
Tomoki Toda and Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology, Japan

Abstract
Speech translation is a technology that help people communicate across different languages. The most commonly used speech translation model is composed by Automatic Speech Recognition (ASR), Machine Translation (MT) and
Text-To-Speech synthesis (TTS) components, in which they are sharing information only in text level. However, spoken communication is diﬀerent fromwritten communication, as we use rich acoustic cues in order to transmit more
information. This paper is concerned with speech-to-speech translation that issensitive to paralinguistic information. Our long-term goal is to made a systemthat allows user to speak a foreign language with the same expressiveness as
if they were speaking in their own language by reconstructing input acoustic
features (F0, duration, spectrum etc.) in the target language. From the many
diﬀerent possible paralinguistic features to handle, in this paper we chose duration and power as a first step, proposing a method that can translate thesefeatures from input speech to the output speech in continuous space. This isdone in a simple and language-independent fashion by training an end-to-end
model that maps source language duration and power information into the target language. Two approaches are investigated including regression and Neural
Network (NN) model. We evaluate the proposed method and show that paralinguistic information in input speech of source language can appears in output
speech of target language.
Keywords: Paralinguistic Information, Speech to speech translation, Emotion

Preprint submitted to Machine Translation Journal

July 15, 2016

,Automatic Speech Recognition, Machine Translation, Text to Speechsynthesis.

1. Introduction
We speak with many diﬀerent varieties of acoustic and visual cues to convey
our thoughts and emotions. Many of those paralinguistic cues transmit additional information that cannot be expressed in words. It may not be a critical5factor in written communication, but in spoken communication it has great importance. Because even if the content of words is the same, if the intonationand facial expression are diﬀerent an utterance can take an entirely diﬀerent
meaning. Therefore it is necessary to take into account paralinguistic factor inany systems that are constructed to augment human-to-human communication.

10

Speech-to-speech translation system is one of technologies that help people
communicate across diﬀerent languages. However, standard speech translation
systems only convey linguistic content from source languages to target languages
without considering paralinguistic information. Although the input of ASR
contains rich prosody information, but the words output by ASR is in written

15

form that have lost all prosody information. The words output by TTS willthen be given the canonical prosody for the input text, not reflecting these
traits. Thus, information sharing between the ASR, MT, and TTS modules isweak, and after ASR source-side acoustic details are lost (for example: speechrhythm, emphasis, or emotion).

20

This paper is concerned with speech-to-speech translation that is sensitive
to paralinguistic information. Our long-term goal is to made a system thatallows user to speak a foreign language with the same expressiveness as if they
were speaking in their own language by reconstructing input acoustic features(F0, duration, spectrum etc.) in the target language. From the many diﬀerent

25

possible paralinguistic features to handle, in this paper we chose duration andpower as a first step, proposing a method that can translate these features from
input speech to the output speech in continuous space.

2

First, we extract features at the level of Hidden Markov Model (HMM)
states, and use linear regression to translate them to the duration and power of30

HMM states of the output speech. Furthermore, we also expand the paralinguistic translation model to adapt to more general tasks by training a single modelthat is applicable to all words using neural networks. There are two merits tousing neural networks. First, neural network possess suﬃcient power to express
diﬃcult regression problems such as translation of acoustic features for multiple

35

words. Second, neural network can be expanded with features expressing additional information such as the input word and translated word, the position ofboth words, parts of speech, and so on. We perform experiments that use thistechnique to translate paralinguistic features and reconstruct the input speech’s
paralinguistic information, particularly emphasis, in output speech.

40

2. Conventional Speech-to-Speech Translation
In Conventional Speech-to-Speech, ASR module decode text of utterance
from input speech. Now acoustic feature represent as X = [x1 , x2 . . . xT ] andspoken word represent as E = [f1 f2 . . . fN ] then the probability is P (E | X).
ASR system decode E that maximize P (E | X). P (E | X) can convert by
Bayes’ theorem as below
P (E | X) =P (X | E)P (E)
P (X)

(1)

From point of E view P (X) is a constant value. We can covert equation as
P (E | X) ∝ P (X | E)P (E)(2)

Then P (X | E) is Acoustic Model(AM) and P (E) is Language Model(LM).
MT module decode target words sequence J that maximize probability
P (J |E) given E.
Ĵ = argmax P(J |E)(3)

As same as ASR we can convert P (J |E) as below.
Ĵ ∝ argmax

P (E|J )P (J )
P (E)

3

(4)

Then P (E|J ) is a translation model.
TTS module generate speech parameter O = o1 , o2 , . . . oT , T is the lengthof O, given HMM AM λ = [λ1 , λ2 . . . λN ] that represent J. The out put O =o1 , o2 , . . . oT can be represent byÔ = argmaxO P(O | λ, T)(5)

In these three module, they share information as E or J , so that the input
non-lexical information of E lost through ASR.

45

3. Speech translation considering paralinguistic information
For achieve paralinguistic information translation, we need consider how tohandle paralinguistic features such as X. In ASR and TTS module, phoneme
is a smallest lexical unit that represent speech. And in MT module, a word isa smallest unit of system. From point of speech processing phoneme is a good50

segment to handle paralinguistic features but in human speaking we usually
speak emotionally such as emphasis, surprise and sadness in word, phrase andsentence level. So we consider word is better segment to learn these prosody
mapping rule between source to target speech. We make the word AM foreach word and perform ASR and TTS. Then we extract the acoustic features

55

X belong to each words and translate X through ASR and translate acoustic
feature from source to target directory by regression model in MT part. Finally
we use translated acoustic features to generate output speech’s in TTS part.
3.1. Speech Recognition
The first step of the process uses ASR to recognize the lexical and paralinguistic features of the input speech. This can be represented formally as
E , X |S),ÊÊ, X̂ = argmax P (E

(6)

X
E ,X

where S indicates the input speech, E indicates the words included in the ut60

terance and X indicates paralinguistic features of the words in E . In order to4

Figure 1: Overview of proposed method5recognize this information, we construct a word-based HMM acoustic model.
The acoustic model is trained with audio recordings of speech and the corresponding transcriptions E using the standard Baum-Welch algorithm. Once wehave created our model, we perform simple speech recognition using the HMM
65

acoustic model and a language model that assigns a uniform probability to all
digits. Viterbi decoding can be used to find E. Finally we can decide the duration vector xi of each word ei based on the time spent in each state of the
HMM acoustic model in the path found by the Viterbi algorithm. The powercomponent of the vector is chosen in a similar way, and by taking the mean

70

power value of each feature over frames that are aligned to the same state of theacoustic model. We express power as [power, ∆power, ∆∆power] and join thesefeatures together as a super vector to control power in the translation step. In
ASR part, we don’t need labeling the prosody of speech we just segment eachwords and extract observed acrostic feature.

75

3.2. Lexical Translation
Lexical translation finds the best translation J of sentence E .
J |E
E ),
Jˆ = argmax P (J(7)

J

where J indicates the target language sentence and E indicates the recognizedsource language sentence. Generally we can use a statistical machine translation,
to obtain this translation in standard translation tasks, but for digit translation
we can simply write one-to-one lexical translation rules with no loss in accuracy.
80

3.3. Paralinguistic Translation
Paralinguistic translation converts the source-side acoustic features vector
X into the target-side acoustic features vector Y according to the following
equationY |X
X ).
Ŷ = argmax P (Y(8)

Y

In particular, we control duration and power of each word using a source-side⊤

x1 , · · · , xNx ] and a target-side duration
duration and power super vector xi = [x6]⊤[and power super vector yi = y1 , · · · , yNy . In these vectors Nx represents
the number of HMM states on the source side and Ny represents the number
85

of HMM states on the target side. The sentence duration and power vector
consists of the concatenation of the word duration and power vectors such thatY = [yy1 , · · · , yi , · · · , yI ] where I is the length of the sentence. We can assume
that duration and power translation of each word pair is independent from thatof other words, allowing us to find the optimal Y using the following equation:Ŷ = argmax

∏

Y
90

xi ).
P (yyi |x(9)ixi ) is defined according
The word-to-word acoustic translation probability P (yyi |x
to linear regression matrix that indicates that yi is distributed according to anormal distributionxi ) = N (yyi ; W ei ,ji x′i , S)
P (yyi |x

(10)

[
]⊤
where x′ is 1x⊤ and W ei ,ji is a regression matrix (including a bias) defining alinear transformation expressing the relationship in duration and power between
ei and ji . An important point here is how to construct regression matrices
for each of the word pairs ⟨e, j⟩ we want to translate. In order to do so, we
optimize each regression matrix on the translation model training data for ⟨e, j⟩
by minimize root mean squared error (RMSE) with a regularization termŴ e,j = argmin
W e,j

N
∑

W e,j ||2 ,||yy∗ n − y n ||2 + α||W

(11)

n=1

where N is the number of training samples for the word pair, n is the ID of eachtraining sample, y ∗ is target language reference word duration and power vector,
95

and α is a hyper-parameter for the regularization term to prevent over-fitting.1
This maximization can be solved in closed form using simple matrix operations.

1 We chose α to be 10 based on preliminary tests but the value had little eﬀect on subjective

results.

7

3.4. Speech Synthesis
In the TTS part of the system we use an HMM-based speech synthesis
system [1], and reflect the duration and power information of the target word
100

paralinguistic information vector onto the output speech. The output speechparameter vector sequence C = [cc1 , · · · , c T ]⊤ is determined by maximizing thetarget HMM likelihood function given the target word duration and power vectorŶ and the target language sentence Jˆ as follows:ˆ Ŷ )O |J
Ĉ = argmax P (O
J,

(12)

C

subject to O = M C ,(13)

where O is a joint static and dynamic feature vector sequence of the target
105

speech parameters and M is a transformation matrix from the static feature
vector sequence into the joint static and dynamic feature vector sequence. While
TTS generally uses phoneme-based HMM models, we instead used a word based
HMM to maintain the consistency of feature extraction and translation. In thistask the vocabulary is small, so we construct an independent context model.

110

4. End to End models of Paralinguistic feature translation methods
In this section we describe two ways to translate paralinguistic features ofthe source words to target words. One is simple linear regression another is
Neural Network with word embed vector.
4.1. Linear regression models
Paralinguistic translation converts the source-side paralinguistic feature X
into the target-side paralinguistic feature Y flowing Voice Conversion ideas [2,3, 4]Ŷ = argmax P (Y|X)

(14)

Y
115

In particular, we control duration and power using source-side word feature vec[]⊤⊤
tor xi = [x1 , · · · , xNx ] and target-side word feature vector yi = y1 , · · · , yNy .
In these vectors Nx represents the number of HMM states in source side and8Ny represents the number of HMM states in target side. The sentence feature
vector consists of the concatenation of the word duration and power vectors such120

as Y = [y1 , · · · , yi , · · · , yI ] where I is the length of the sentence. We assume
that duration and power translation of each word pair is independent, giving
the following equation:Ŷ = argmax∏

Y

P (yi |xi ).

(15)

i

This can be defined with any function, but we choose to use linear regression,
which indicates that yi is distributed according to a normal distribution
P (yi |xi ) = N (yi ; Wei ,ji x′i , S)
125

(16)

[
]⊤
where, x′ is 1x⊤ and Wei ,ji is a regression matrix with bias defining a linear
transformation expressing the relationship in duration and power between ei
and ji . An important point here is how to construct regression matrices foreach of the words we want to translate. In order to do so, we optimize each
regression matrix on the translation model training data by minimize root mean

130

squared error (RMSE) with a regularization termŴe,j = argmax

N∑

Wei ,ji n=1||y ∗ n − y n ||2 + α||Wei ,ji ||2 ,(17)

where N is the number of training samples, n is an id of a training sample,y ∗ is target language reference word duration and power vector, and α is a
hyper-parameter for the regularization term.2
Linear regression model need train a regression matrix for each word pair
135⟨e, j⟩. The simplest way to generalize this model is by not training a separate
model for each word, but a global model for all words in the vocabulary. Thiscan be done by changing the word-dependent regression matrix W e,j into asingle global regression matrix W and training the matrix over all samples in

2 We chose α to be 10 based on preliminary tests.

9the corpus. However, this model can be expected to not be expressive enough140

to perform paralinguistic translation properly. For example, the mapping ofduration and power from a one-syllable word to another one-syllable word, andfrom a one-syllable word to a two-syllable word would vary greatly, but thelinear regression model only has the power to perform the same mapping foreach word.

145

4.2. Global Neural Network Models
As a solution to the problem of the lack of expressibility in linear regression,
we propose a global method for paralinguistic translation using neural networks.
Neural networks have higher expressive power due to their ability to handle
non-linear mappings, and are thus an ideal candidate for the task. In addition,

150

they allow for adding features for many diﬀerent types of information following
ASR, MT and TTS’s common practice, such as word ID vectors, word position,left and right words of input and target words, part of speech, the number ofsyllables, accent types, etc. This information is known to be useful in TTS [1],so we can likely improve estimation of the output duration and power vector in

155

translation as well.
In this research, we prepare a feed forward neural network that proposes thebest output word acoustic feature vector Ŷ given input word acoustic feature
vector X . As additional features, we also add a binary vector with the ID of
the present word set to 1, and the position of the output word. In this work,160

because the task is simple we just use this simple feature set, but this could be
expanded easily more for complicated tasks.
For the sake of simplicity in this formulation we show an example with theword acoustic feature vector only. First, we set each input unit ιi equal to the
input vector value:ιi = x i .

(18)

The hidden units πj are calculated according to the input-hidden unit weight10

Figure 2: Neural Network for acoustic feature translation

matrix W h :
πj =

1
∑ h ,
1 + exp(−α i wij
ιi )

(19)

where α is gradient of sigmoid function. The output units ψk and final acoustic
feature output yk are set as
ψk

=

∑

o
wjk
πj

(20)

j

yk

=

ψk ,

(21)

where W o is hidden-output unit weight matrix. As an optimization criterion we165

use minimization of RMSE, which is achieved through simple back propagation.

5. Evaluation
5.1. Experimental Setting
We examine the eﬀectiveness of the proposed method through EnglishJapanese speech-to-speech translation experiments, summarized in Table 1. In
170

these experiments we assume the use of speech-to-speech translation in a situation where the speaker is attempting to reserve a ticket by phone in a diﬀerent
language. When the listener makes a mistake when listening to the ticket number, the speaker re-speaks, emphasizing the mistaken number. In this situation,
if we can translate the paralinguistic information, particularly emphasis, this175

will provide useful information to the listener about where the mistake is. In
11

ASR
Tool

HTK

Training sentences

8440

HMM states

16

MT
Tool

Moses

Training utterances

445

Test utterances

55

Neural net structure

29/25/16

TTS
Tool

HTS

Training utterances

445

HMM states

16

Table 1: Experimental Settingsorder to simulate this situation, we recorded a bilingual speech corpus where
an English-Japanese bilingual speaker emphasizes one word during speech in astring of digits. The lexical content to be spoken was 500 sentences from the
AURORA2 data set, chosen to be word balanced by greedy search [5]. The
180

training set is 445 utterances and the test set is 55 utterances.3
We further used this data to build an English-Japanese speech translation
system that include our proposed paralinguistic translation model. We used the
AURORA2 8440 utterance bilingual speech corpus to train the ASR module.
Speech signals were sampled at 8kHz with utterances from 55 males and 55

185

females. We set the number of HMM states per word in the ASR acoustic
model to 16, the shift length to 5ms, and other various settings for ASR to follow[6][7]. The original Aurora2 has 8440 utterances for training and 4004 utterances
for testing. Here we don’t use the original testing part in our experiments.

3 Freely available at http://www.phontron.com/pcbue

12Base line V.S. Linear Regression
None

No translation of paralinguistic information

EachLR

Linear regression with a model for each word

Table 2: Compare baseline against prosed linear regression for each words

Proposed LR V.S. NN model
AllLR

A single linear regression model trained on all words

AllNN

A single neural network model trained on all words

AllNN -ID

The AllNN model without additional features

Table 3: Compare proposed Linear Regression against Neural Network model for all words

To simplify the problems, the experiments were done where the ASR has no190

error. Therefore we selected 500 balanced sentence from the 8440 utterances oftraining data, and divide the utterances into 445 utts for training set and 55
utts of testing for the paralinguistic translation. So the achieve 100% accuracy
is because it is a closed test set. For TTS, we use the same 445 utterances
for training an independent context synthesis model. In this case, the speech

195signals were sampled at 16kHz. The shift length and HMM states are identicalto the setting for ASR.
In the evaluation, we compare the following two baselines Table 2 and proposed three global models of paralinguistic translation each other Table 3.
In addition, we use naturally spoken speech as an oracle output.

200

5.2. Objective Evaluation
We first perform an objective assessment of the translation accuracy of duration and power, the results of which are found in Figure 3, 4, 5 and Figure
6. We compared the diﬀerence between the system duration and power and thereference speech duration and power in terms of RMSE.

205

From these results, we can see that the AllLR model is not eﬀective at
mapping duration and power information, achieving results largely equal to the13

Figure 3: Root mean squared error rate (RMSE) between the reference target duration andthe system output for each digit

Figure 4: Root mean squared error rate (RMSE) between the reference target power and thesystem output for each digit

baseline. The AllNN model without linguistic information does slightly betterbut still falls well short of the EachNN baseline. Finally, we can see that AllNN
is able to eﬀectively model translation of paralinguistic information, although210

accuracy of power lags slightly behind that of duration.

14

Figure 5: RMSE between the reference and system duration

Figure 6: RMSE between the reference and system power

We also show the relationship between the number of NN hidden units and
RMSE of duration in 7 (the graph for power was similar). It can be seen that
RMSE continues to decrease as we add more units, but with diminishing returns
after 25 hidden units. When comparing the number of free parameters in the
215

EachLR model (17*16*11=2992) and the AllNN model with 25 hidden units(28*25+25*16=1100), it can be seen that we were able to significantly decrease15

Figure 7: RMSE of duration for each number of NN hidden units

Figure 8: Prediction rate

the number of parameters with little change in accuracy.
5.3. Subjective Evaluation
As a subjective evaluation we asked native speakers of Japanese to evalu220

ate how well emphasis was translated into the target language for the baseline, oracle, and EachLR and AllNN models when translating duration or duration+power.
The first experiment asked the evaluators to attempt to recognize the identi-

16

Figure 9: Subjective degree of emphasisties and positions of the emphasized words in the output speech. The overview
225of the result for the word and emphasis recognition rates is shown in Figure
8. We can see that all of the paralinguistic translation systems show a clear
improvement in the emphasis recognition rate over the baseline. There is no
significant diﬀerence between Linear Regression and Neural Network model, indicating that the neural network learned a paralinguistic information mapping

230

that allows listeners to identify emphasis eﬀectively.
The second experiment asked the evaluators to subjectively judge the strengthof emphasis with the following three degrees:
1: not emphasized
2: slightly emphasized235

3: emphasized
The overview of the experiment regarding the strength of emphasis is shown in
Figure 9. This figure shows that all systems show a significant improvementin the subjective perception of strength of emphasis. In this case, there seems
to be a slight subjective preference towards EachLR when power is considered,240

reflecting the slightly larger RMSE found in the automatic evaluation. We

17also performed emphasis translation only use power, but generated speech’s
naturalness is quiet low. There are the value of speech volume drastic changes
in short time. Because of our proposed method process we extract power featurefor each flames that given by duration information so that the power extraction
245

has high dependency on duration. In this method, if we try to handle otheracoustic feature(example F0) then we need to modeling duration together.

6. Related Works
There have been several studies demonstrating improved speech translation
performance by translation source side speech non-lexical information to target
250

side speech non-lexical information. In previous work, [8, 9, 10] focus for input
speech’s acoustic information and try to explores a tight coupling of ASR and
MT for speech translation with sharing information, they boost translation quality as measured by BLEU score. Other related works focus on speech intonation
recognizing using that to reduce translation ambiguity on target side[11, 12].

255

These methods consider non-lexical information to boost translation accuracy.
However as we mentioned before, there is more to speech translation than justaccuracy. We should consider other features such as the speakers facial andspeech expressions. There is some research that considers translating these expressions and improves speech translation quality in other ways that cannot

260

be measured by BLEU. For example some work focuses on mouth shape andtries to translate speaker emotion from source to target.[13, 14]. On the other
hand, [15, 16] focus input speech’s prosody, extracting F0 from source speech at
the sentence level and clustering accent groups.These are then translated into
target side accent groups.consider the prosody encoded as factors in a factored265

translation model [17] to convey prosody from source to target.
In our work, we focus on source speech acoustic features and extract themand translate to target acoustic features directly and continuously. In thisframe work, we need two translation models. One for word to word translation
model, another for acoustic translation model. We made acoustic translation

18

270

model with linear regression for each translation pare. In this work our proposed method can translate acoustic feature with consider the similarity andcontinuously. This method is very simple and we can translate acoustic feature(duration) without decline BLEU score.
After this work, there are published related work that modeling emphasis by275

HMM acoustic model and calculate emphasis level and translation the emphasis
level in word level[18, 19]. And this method can handle power and duration
value continuously but for make emphasis acoustic model they need to annotate
speech. We often have a diﬀerent impression even we hear same speech. Sothat annotate for various prosody and emotion sometime not stable because it

280

depends human feeling, In these related works, when they apply their method for
multi-prosody and emotion their will be faced para-speech data collection issue.
But this work did not need any annotate for speech, in point of data preparation
view we can easily adapt our method for multi-prosody and emotion.

7. Conclusion
285

In this paper we proposed a generalized model to translate duration andpower information for speech-to-speech translation. Experimental results showed
proposed method can modeling input speech emphasis more than baseline. But
we also see if we failed to regression target speech feature then we generate worth
quality of speech. This issue can be say machine translation error of paralin-290

guistic information. We should care this issue by make threshold or modelingthe naturalness of paralinguistic feature like lexical MT.
In future work we plan to expand beyond the digit translation task in thecurrent paper to a more general translation task using phrase-based SMT. The
diﬃculty here is the procurement of parallel corpora with similar paralinguistic295

information for large-vocabulary translation tasks. We are currently considering possibilities including simultaneous interpretation corpora and movie dubs.
Another avenue for future work is to expand to other acoustic features such as
F0, which play an important part in other language pairs.

19

8. Acknowledge
300

Part of this research was supported by JSPS KAKENHI Grant Number
24240032 and 26870371.

References[1] H. Zen, K. Tokuda, A. W. Black, Statistical parametric speech synthesis,
Speech Communication.
305[2] T.Toda, A. .W.Black, K.Tokuda, Voice conversion based on maximum
likelihood estimation of spectral parameter trajectory, IEEE Trans. ASLP
15 (8) (2007) 2222–2235.
[3] M.Abe, S.Nakamura, K.Shikano, H.Kuwabara, Voice conversion throughvector quantization, J. Acoust. Soc. Jpn (E) 11 (2) (1990) 71–76.

310[4] Y.Stylianou, O.Cappé, E.Moulines, Continuous probabilistic transform forvoice conversion, IEEE Trans. SAP 6 (2) (1998) 131–142.
[5] J. Zhang, S. Nakamura, An eﬃcient algorithm to search for a minimum
sentence set for collecting speech database, in: Proceedings of the 15th
International Congress of Phonetic Sciences, 2003.

315[6] H. G. Hirsh, D. Pearce, The AURORA experimental framework for the performance evaluations of speech recognition systems under noisy conditions,in: ISCA ITRW ASR2000, 2000.
[7] R. Leonard, A database for speaker independent digit recognition, in: Proceedings of ICASSP, 1984.

320[8] J. Jiang, Z. Ahmed, J. Carson-Berndsen, P. Cahill, A. Way, Phonetic
representation- based speech translation, in: Machine Translation Summit,
2011.

20

[9] M. Ohgushi, G. Neubig, S. Sakti, T. Toda, S. Nakamura, An empiricalcomparison of joint optimization techniques for speech translation, in: 14th325

Annual Conference of the International Speech Communication Association(InterSpeech 2013), Lyon, France, 2013, pp. 2619–2622.
[10] M. Dreyer, Y. Dong, Apro: All-pairs ranking optimization for mt tuning,in: Proceedings of the 2015 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Language Technolo-

330

gies, Association for Computational Linguistics, Denver, Colorado, 2015,
pp. 1018–1023.
URL http://www.aclweb.org/anthology/N15-1106[11] Y. S. N. C. H. I. F. S. A. Y. Toshiyuki Takezawa, Tsuyoshi Morimoto, S. Yamamoto, A japanese-to- english speech translation system: Atr-matrix, in:335

In Proceedings of the 5th International Conference on Spoken Language
Processing, ICSLP, Sydney,Australia, 1998, p. pp. 957960.
[12] W. Wahlster, Robust translation of spontaneous speech: a multi-engine approach, in: IJCAI’01 Proceedings of the 17th international joint conferenceon Artificial intelligence - Vol 2, IJCAI, Washington,America, 2001, pp. p

340

1484–1493.
[13] S. N. Shin Ogata, Takafumi Misawa, S. Morishima, Muti- modal translation system by using automatic facial image tracking and model- based lip
synchronization, in: ACM SIGGRAPH2001 Conference Abstracts and Applications,Sketch and Applications, Siggraph, Los Angeles, 2001, p. p231.

345[14] S. Nakamura, Y. Sasaki, Toward common evaluation frameworks for speech
recognition and speech translation technologies, The Journal of the Acoustical Society of Japan.
[15] P. D. Agüero, J. Adell, A. Bonafonte, Prosody generation for speech-tospeech translation, in: Proceedings of ICASSP, 2006.

21

350

[16] V. Kumar, S. Bangalore, S. Narayanan, Enriching machine-mediatedspeech-to-speech translation using contextual information, Computer
Speech and Language.
[17] P. Koehn, H. Hoang, Factored translation models, in: Proceedings of
EMNLP, 2007, pp. 868–876.

355[18] Q. T. Do, S. Sakti, G. Neubig, T. Toda, S. Nakamura, Improving translation of emphasis with pause prediction in speech-to-speech translation
systems, in: 12th International Workshop on Spoken Language Translation (IWSLT), Da Nang, Vietnam, 2015.
[19] Q. T. Do, S. Takamichi, S. Sakti, G. Neubig, T. Toda, S. Nakamura, Pre-360

serving word-level emphasis in speech-to-speech translation using linear regression HSMMs, in: 16th Annual Conference of the International SpeechCommunication Association (InterSpeech 2015), Dresden, Germany, 2015.

22