An End-to-end Model for Cross-Lingual Transformationof Paralinguistic Information
Takatomo Kano, Shinnosuke Takamichi, Sakriani Sakti, Graham Neubig,
Tomoki Toda, and Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology, Japan

Abstract
Speech translation is a technology that helps people communicate acrossdifferent languages. The most commonly used speech translation model iscomposed of Automatic Speech Recognition (ASR), Machine Translation(MT) and Text-To-Speech synthesis (TTS) components, which share information only at the text level. However, spoken communication is different
from written communication in that it uses rich acoustic cues such as prosody
in order to transmit more information through non-verbal channels. This
paper is concerned with speech-to-speech translation that is sensitive to thisparalinguistic information. Our long-term goal is to make a system that allows users to speak a foreign language with the same expressiveness as if they
were speaking in their own language. Our method works by reconstructing
input acoustic features in the target language. From the many different possible paralinguistic features to handle, in this paper we chose duration andpower as a first step, proposing a method that can translate these featuresfrom input speech to the output speech in continuous space. This is done ina simple and language-independent fashion by training an end-to-end modelthat maps source language duration and power information into the target
language. Two approaches are investigated: linear regression and neural network models. We evaluate the proposed method and show that paralinguistic
information in the input speech of the source language can be reflected in
the output speech of the target language.
Keywords: Paralinguistic Information, Speech to speech translation,
Emotion, Automatic Speech Recognition, Machine Translation, Text to
Speech synthesis

Preprint submitted to Machine Translation Journal

April 10, 2018

1. Introduction

5

10

15

20

25

30

35

When we speak, we use many different varieties of acoustic and visualcues to convey our thoughts and emotions. Many of those paralinguistic cues
transmit additional information that cannot be expressed in words. Whilethese cues may not be a critical factor in written communication, in spoken
communication they have great importance; even if the content of the words
are the same, if the intonation and facial expression are different an utterance
can take an entirely different meaning. As a result, it would be advantageous
to take into account these paralinguistic features of speech in any system thatis constructed to aid or augment human-to-human communication.
Speech-to-speech translation helps people communicate across different
languages, and is thus one prime example of such a system. However, standard speech translation systems only convey linguistic content from source
languages to target languages without considering paralinguistic information. Although the input of ASR contains rich prosody information, thewords output by ASR are in written form that have no indication of the
prosody included in the original speech. As a result, the words output by
TTS on the target side will thus be given the canonical prosody for the input
text, not reflecting the prosodic traits of the original speech. In other words,
because information sharing between the ASR, MT, and TTS modules islimited to only lexical information, after the ASR conversion from speech totext, source-side acoustic details such as rhythm, emphasis, or emotion are
lost.
This paper is concerned with speech-to-speech translation that is sensitive
to paralinguistic information, with the long-term goal of making a systemthat allows a user to speak a foreign language with the same expressivenessas if they were speaking in their own language. The proposed method works
by recognizing acoustic features (duration and power) in the source language,
then reconstructing them in the target language. From the many different
possible paralinguistic features to handle, in this paper we chose duration andpower as a first step, proposing a method that can translate these featuresfrom the input speech to the output speech in continuous space.
First, we extract features at the level of Hidden Markov Model (HMM)
states, the use a paralinguistic translation model to predict the duration andpower features of HMM states of the output speech. Specifically, we use
two approaches: a linear regression model that predicts separately predicts
prosody for each word in the vocabulary, and a model that can adapt to more2

general tasks by training a single model that is applicable to all words in the
vocabulary using neural networks.140

45

2. Conventional Speech-to-Speech Translation
In conventional speech-to-speech translation systems, the ASR module
decodes the text of the utterance from input speech. Acoustic features are
represented as A = [a1 , a2 ...aNa ] and the corresponding words are represented
as E = [e1 , e2 , ..., eNe ]. Na and Ne are the lengths of the acoustic feature
vectors and spoken words respectively.
The ASR system finds E that maximizes P (E|A). By Bayes’ theorem,
we can convert this toP (E|A) ∝ P (A|E)P (E),

50

(1)

where P (A|E) is the Acoustic Model (AM) and P (E) is the Language Model(LM). The MT module finds the target words sequence J that maximizes
probability P (J |E):Jˆ = argmax P (J |E).
(2)
J

Similarly to what was done for ASR, we can convert P (J |E) as follows:Jˆ = argmax P (E|J )P (J ),

(3)

J

55

60

where P (E|J ) is a translation model and P (E) is a language model.
The TTS module generates speech parameters O = [o1 , o2 , ..., oNo ] given
HMM AM states Hx = [h1 , h2 , ..., hNh ] that represent J . Here No and Nh
is the length of the generated speech parameter sequence and the number ofstates of the HMM AM. The output O = [o1 , o2, ..., oNo ] can be representedbyÔ = argmax P (O|H)(4)
These three modules, only share information through E or J , which arestrings of text in the source and target languages respectively. As a result,
all non-verbal information that was original expressed in source speech A islost the moment it is converted into source text E by ASR.
1Part of the content of this article is based on content that has been published in
IWSLT and InterSpeech [10, 11]. In this paper describe these methods using an unified
formulation, adds a more complete survey, and discuss the results in significantly more
depth.

3

3. Speech Translation considering Paralinguistic Information

65

70

75

80

85

90

95

In order to perform speech translation in a way that is also able to consider
paralinguistic information, we need consider how to handle paralinguisticfeatures included in A. Specifically, we need to extract acoustic featuresduring ASR, translate them to another language during MT, and then reflect
them in the target speech during TTS.
The first design decision we need to make is at what granularity at which
to represent paralinguistic features: phoneme, word, phrase, or sentence level.
In the ASR and TTS modules, phonemes are the smallest lexical unit that
represent speech, and in the MT module, words are the smallest unit handled
by the system. From the point of view of speech processing phonemes are agood granularity with which to handle paralinguistic features. However, inhuman speech, paralinguistic features such as emphasis, surprise, and sadness
can be more intuitively attributed to the word, phrase and sentence level [19].
Thus, as the main focus of our work is on methods for translation of emphasis
between languages, for this paper we decide to construct our models purely
on the word level. We create word-level AMs for ASR and TTS, extract theparalinguistic features X belonging to each word, and translate these wordlevel acoustic features from the source to target directly using a regression
model in the MT module. Finally we use translated acoustic features togenerate output speech in the TTS module.
While the overall framework here is independent of the speech translationtask, as the research is ambitious, our experiments below focus on a limited
setting of translating digits. This digit translation task can be motivated bya situation where a customer is contacting a hotel staff member attempting
to make a reservation. The customer conveys the reservation number, andthe hotel staff member confirms, but the number turns out to be incorrect.
In this case, the customer would re-speak the number, using prosody toemphasize the missing information. The problem formulation below will also
use this setting as an example, specifically the example of English-Japanesetranslation.
3.1. Speech Recognition
The first step of the process uses ASR to recognize the lexical and paralinguistic features of the input speech. This can be represented formallyasÊ, X̂ = argmax P (E, X|A),
(5)
E,X

4

100

105

110

115

120

where A indicates the input speech, E indicates the words included in theutterance and X indicates paralinguistic features of the words in E. In
order to recognize this information, we construct a word-based HMM AM.
The AM is trained with audio recordings of speech and the correspondingtranscriptions E using the standard Baum-Welch algorithm. Once we havecreated our model, we perform simple speech recognition using the HMM
AM and a language model that assigns a uniform probability to all digits.
Viterbi decoding can be used to find E. Finally we can decide the duration
vector xi of each word ei based on the time spent in each state of the HMM
AM in the path found by the Viterbi algorithm. The power component of thevector is chosen in a similar way, and by taking the mean power value overframes that are aligned to the same state of the AM. We express power as[power, ∆power, ∆∆power] and join these features together as a super-vector
to control power in the translation step. ∆ indicates dynamic features. It
should be noted that in contrast to other work such as [2], for the ASR part,
we dont need a manual labeling the prosody of speech and simply segmenteach word and extract observed acoustic features.
3.2. Lexical and Paralinguistic Translation
Lexical translation finds the best translation J of recognized source sentence E. Generally we can use any variety of statistical machine translation
to obtain this translation in standard translation tasks, but for digit translation we can simply write one-to-one lexical translation rules with no loss
in accuracy such as ji = ei where i is word index. Paralinguistic translation converts the source-side acoustic feature vector X into the target-side
acoustic feature vector Y according to the following equationŶ = argmax P (Y |X)

(6)

Y

125

130

There are many types of acoustic features used in ASR and TTS systems,
including MFCC, MGC, Filter-bank, F0, power, and duration. In this work
we use power and duration to express “emphasis information”. We makethis decision due to the fact that MFCC, Filter-bank, and MGC features are
more strongly connected to lexical information related to the content of theutterance. F0, power and duration are more correlated with paralinguistic
information regarding the method of speech, but because Japanese is a tonal
language where F0 has a strong relationship with content distinctions, in thiswork we focus on duration and power. We control duration and power of each
5

135

140

word using a source-side duration and power super-vector xi = [x1 , ..., xNx ]and a target-side duration duration and power super-vector yi = [y1 , ..., yNy ].
Here Nx and Ny represent the length of the paralinguistic feature vector foreach word i.
In these vectors Nx represents the number of HMM states on the sourceside and Ny represents the number of HMM states on the target side. The
sentence duration and power vector consists of the concatenation of the word
duration and power vectors such that Y = [y1 , . . . , yn , . . . , yNy ]. We can assume that duration and power translation of each word pair is independent
from that of other words, allowing us to find the optimal Y using the following equation
YŶ = argmax
P (yn |xn )
(7)
Y

n

The word-to-word acoustic translation probability P (yn |xn ) is calculated
according to a linear regression matrix that indicates that yi is distributed
according to a normal distribution
P (yi |xi ) = N (yi ; Wei ,ji , x0i , A)
145

150

(8)

where x0 is transposed x and Wei ,ji is a regression matrix with bias defining
a linear transformation expressing the relationship in duration and power
between ei and ji . An important point here is how to construct regression
matrices for each of the words we want to translate. In order to do so, we
optimize each regression matrix in the translation model training data by
minimizing root mean squared error (RMSE) with a regularization termŴei ,ji = argmax
Wei ,jiNXkyn∗ − yn k2 + αkWei ,ji k2 ,

(9)

n=1

where N is the number of training samples, n is the id of a training sample,y∗ is target language reference word duration and power vector, and α isa hyper-parameter for the regularization term to prevent over-fitting. Thismaximization can be solved in closed form using simple matrix operations.
155

3.3. Speech Synthesis
In the TTS part of the system we use an HMM-based speech synthesis
system [24], and reflect the duration and power information of the target
word paralinguistic information vector onto the output speech.
Ĥy = argmax P (Hy |Y )
6

(10)

160

The output speech parameter vector sequence O = [o1 , ..., oNo ] is determinedby maximizing the target HMM AM Ĥy likelihood function given the target
language sentence Jˆ as follows:Ô = argmax P (C|Jˆ, Ĥy )

(11)

O

subject to C = M O

165

170

(12)

where C is a joint static and dynamic feature vector sequence of the target
speech parameters and M is a transformation matrix from the static feature
vector sequence into the joint static and dynamic feature vector sequence.
When generating speech, the corresponding HMM AM parameters and thelength of the target language state sequence are determined by Ŷ resulting
from the paralinguistic translation step. While TTS generally uses phonemebased HMM models, we instead used a word-based HMM to maintain theconsistency of feature extraction and translation. Usually, in TTS phonemebased HMM AMs, the current HMM AM is heavily influenced by the previous
and next phonemes, making it necessary to consider context information from
input sentence. However, in the digit translation task the vocabulary is small,so we construct an word level independent context HMM AM.
4. End-to-end Paralinguistic Translation Methods

175

180

In this section we describe two ways to translate paralinguistic features ofthe source words to target words. The first is simple linear regression modelthat trains a separate model for each word in the vocabulary, and anotheris neural network model that trains a single model for the entire vocabulary
but provides the model with information of the word identity.
4.1. Linear Regression Models
Paralinguistic translation converts the source-side paralinguistic features
X into the target-side paralinguistic features Y , in a manner inspired byprevious work on voice conversion [1, 21]
Ŷ = argmax

P (Y |X)

(13)

Y

185

In particular, we control duration and power using the source-side word
feature vector xi = [x1 , ..., xNh ] and target-side word feature vector yi =[y1 , ..., yNh ]. Here i represents the word id within the vocabulary. In these7Figure 1: Overview of the proposed method

190

vectors Nh represents the number of HMM states on the source and target sides. The sentence feature vector consists of the concatenation of theword duration and power vectors such as Y where I is the length of thesentence. We assume that duration and power translation of each word pair8

is independent, giving the following equation:Y
P (yi |xi )Ŷ = argmax

(14)

Y

This can be defined with any function, but we choose to use linear regression,
which indicates that yi is distributed according to a normal distribution
P (yi |xi ) = N (yi ; Wei ,ji , x0i , S)

195

200

205

210

215

220

(15)

where, x0 is transposed x and Wei ,ji is a regression matrix with bias defining
a linear transformation expressing the relationship in duration and power
between ei and ji .
An important point here is how to construct regression matrices for eachof the words we want to translate. In order to do so, we optimize each
regression matrix on the translation model training data by minimize RMSE
with a regularization term. This separate training of a model for each wordpair allows the model to be expressive enough to learn how each words’
acoustics are translated into the target language. However, this has serious
problems with generalization, as we will not be able to translate any words
that have not been observed in our training data a sufficient number of times
to learn the transformation matrix. The simplest way to generalize thismodel is by not training a separate model for each word, but a global model
for all words in the vocabulary. This can be done by changing the worddependent regression matrix Wei ,ji into a single global regression matrix
W and training the matrix over all samples in the corpus. However, thismodel can be expected to not be expressive enough to perform paralinguistictranslation properly. For example, the mapping of duration and power froma one-syllable word to another one-syllable word, and from a one-syllable
word to a two-syllable word would vary greatly, but the linear regression
model only has the power to perform the same mapping for each word.
4.2. Global Neural Network Models
As a solution to the problem of the lack of expressiveness in linear regression, we additionally propose a global method for paralinguistic translation
using neural networks. Neural networks have higher expressive power due totheir ability to handle non-linear mappings, and are thus an ideal candidate
for this task. In addition, they allow for adding features for many different
types of information following ASR, MT, and TTSs common practice, such
9Figure 2: Neural Network for acoustic feature translation

225

230

235

as word ID vectors, word position, left and right words of input and target words, part of speech, the number of syllables, accent types, etc. This
information is known to be useful in TTS [24], so we can likely improve estimation of the output duration and power vector in translation as well. In
this research, we use a feed forward neural network that proposes the best
output word acoustic feature vector given input word acoustic feature vector
X. As additional features, we also add a binary vector with the ID of thepresent word set to 1, and the position of the output word. In this work,
because the task is simple we just use this simple feature set, but this could
be expanded easily more for complicated tasks. For the sake of simplicity inthis formulation we show an example with the word acoustic feature vector
only. First, we set each input unit xi equal to the input vector value: li = xi
The hidden units hj are calculated according to the input-hidden unit weightmatrix Wh :
1
P h
(16)
πj =
1 + exp(−α i wi,jli )where α is gradient of sigmoid function. The output units ψk and final
acoustic feature output yk are set as
X
o
ψk =
wj,k
πj .yk = ψk(17)jwhere Wo is the hidden-output unit weight matrix. As an optimization criterion we use minimization of RMSE, which is achieved through simple back
10

240

propagation and weight update, as is standard practice in neural networkmodels.
5. Evaluation

245

250

255

260

265

270

5.1. Experimental Setting
We examine the effectiveness of the proposed method through EnglishJapanese speech-to-speech translation experiments. We use the “AURORA2” data set. The “AURORA-2” data are based on a version of the original
TIDigits down-sampled at 8 kHz from 55 male and 55 female speakers. Different noise signals have been artificially added to clean speech data.
As mentioned previously, in these experiments we assume the use ofspeech-to-speech translation in a situation where the speaker is attempting toreserve a ticket by phone in a different language. When the listener makes a
mistake when listening to the ticket digit, the speaker re-speaks, emphasizing
the mistaken digit. In this situation, if we can translate the paralinguistic
information, particularly emphasis, this will provide useful information to
the listener about where the mistake is. In order to simulate this situation,
we recorded a bilingual speech corpus where an English-Japanese bilingual
speaker emphasizes one word during speech in a string of digits. The content
spoken was 500 sentences from the AURORA-2 test set, chosen to be wordbalanced by greedy search [25] This was further split into a training set of
445 utterances and the test set is 55 utterances.
To train the ASR model, we use 8440 utterances of clean and noisy speechfrom the training set of the AURORA-2 dataset and train with the HTK
toolkit. In the ASR module we trained an HMM AM, where each word has
16 HMM states, and for silence we allocate 3 states. The lexical translationis performed by Moses [13].We further used the 445 utterances of training
data to build an English-Japanese speech translation system that includes
our proposed paralinguistic translation model. We set the number of HMM
states per word in the ASR AM to 16, the shift length to 5ms, and othervarious settings to follow [17, 14]. To simplify the problem, experiments weredone where ASR has no errors. For TTS, we use the same 445 utterances fortraining an independent context synthesis model. In this case, the speech signals were sampled at 16kHz. The shift length and HMM states are identicalto the setting for ASR.
In the evaluation, we compare the following systems

11

275

• Baseline: No translation of paralinguistic information• EachLR: Linear regression with a model for each word
• AllLR: A single linear regression model trained on all words• AllNN: A single neural network model trained on all words• AllNN-ID: The AllNN model without additional features
In addition, we use naturally spoken speech as an oracle output.

Figure 3: Root mean squared error rate (RMSE) between the reference target durationand the system output for each digit
280

285

5.2. Objective Evaluation
We first perform an objective assessment of the translation accuracy of
duration and power, the results of which are found in Figure 3 and 4. For
each of the nine digits plus “oh” and “zero,” we compared the difference between the proposed and baseline duration and power and the reference speech
duration and power in terms of RMSE. From these results, we can see that12

Figure 4: Root mean squared error rate (RMSE) between the reference target power andthe system output for each digit

Figure 5: RMSE between the reference and system duration

13

Figure 6: RMSE between the reference and system power

Figure 7: RMSE of duration for each number of NN hidden units

290

the target speech duration and power output by the proposed method is moresimilar to the reference than the baseline over all eleven categories, indicating the proposed method is objectively more accurate in translating durationand power. Second we compare the proposed linear regression against theneural network model in Figure 5 and 6. We compared the difference be14

295

300

305

310

315

320

tween the system duration and power and the reference speech duration andpower in terms of RMSE. From these results, we can see that the AllLR
model is not effective at mapping duration and power information, achieving
results largely equal to the baseline. The AllNN model without linguistic
information does slightly better but still falls well short of the EachNN baseline. Finally, we can see that our proposed methods outperform baseline and
AllNN is able to effectively model translation of paralinguistic information,although accuracy of power lags slightly behind that of duration.
We also show the relationship between the number of NN hidden units
and RMSE of duration in Figure 7 (the graph for power was similar). It canbe seen that RMSE continues to decrease as we add more units, but withdiminishing returns after 25 hidden units. When comparing the number offree parameters in the EachLR model (17*16*11=2992) and the AllNN modelwith 25 hidden units (28*25+25*16=1100), it can be seen that we were able
to significantly decrease the number of parameters as well.
5.3. Subjective Evaluation
As a subjective evaluation we asked native speakers of Japanese to evaluate how well emphasis was translated into the target language for the baseline, oracle, and EachLR and AllNN models when translating duration or
duration+power. The first experiment asked the evaluators to attempt torecognize the identities and positions of the emphasized words in the output
speech. The overview of the result for the word and emphasis recognition
rates is shown in Figure 8. We can see that all of the paralinguistic translation systems show a clear improvement in the emphasis recognition rate overthe baseline. There is no significant difference between the linear regression
and neural network models, indicating that the neural network learned aparalinguistic information mapping that allows listeners to identify emphasis
effectively. The second experiment asked the evaluators to subjectively judge
the strength of emphasis with the following three degrees:• 1: not emphasized• 2: slightly emphasized• 3: emphasized325

The overview of the experiment regarding the strength of emphasis isshown in Figure 9. This figure shows that all systems show a significant improvement in the subjective perception of strength of emphasis. In this case,15

Figure 8: Prediction rate

Figure 9: Prediction strength of emphasis

330

there seems to be a slight subjective preference towards EachLR when power
is considered, reflecting the slightly smaller RMSE found in the automatic
evaluation. We also performed emphasis translation that only used power,
but the generated speech’s naturalness was quite low. This resulted in drastic speech volume changes in a short time. Because our proposed method
16

335

extracts power features for each frame given by duration information, thepower extraction has a high dependency on duration. In this method, if wetry to handle other acoustic features (e.g. F0) then we also suspect that wewill need to model duration together with these features as well.
6. Related Work

340

345

350

355

360

365

There have been several studies demonstrating improved speech translation performance by translating source side speech non-lexical information
to target side speech non-lexical information. Some previous work [9, 16, 7]
has focused on the input speech information (for example, phoneme similarity, number of fillers, and ASR parameters) and tried to explore a tightcoupling of ASR and MT for speech translation, boosting translation quality as measured by BLEU score. Other related works focus on recognizing
speech intonation to reduce translation ambiguity on the target side [20, 22].
These methods consider non-lexical information to boost translation accuracy. However as we mentioned before, there is more to speech translation than just accuracy, and we should consider other features such as the
speaker’s facial and prosodic expressions.
There is some research that considers translating these expressions andimproves speech translation quality in other ways that cannot be measured
by BLEU. For example some work focuses on facial information and tries totranslate speaker emotion from source to target [19, 15]. On the other hand,[2, 18, 3] focus on the input speechs prosody, extracting F0 from source speechat the sentence level and clustering accent groups. These are then translated
into target side accent groups, considering the prosody encoded as factors ina factored translation model [12] to convey prosody from source to target.
In our work, we focus on source speech acoustic features and extract themand translate to target acoustic features directly and continuously. In thisframework, we need two translation models. One for the word-to-word translation, and another for acoustic translation. We made acoustic translation
models with linear regression for each translation pair. This method is simple, and we can translate acoustic features without having an adverse affecton BLEU score. After this work was originally performed, several related
works have modeled emphasis by HMM AMs and calculated emphasis levels
and translated the emphasis at the word level [5, 6]. These works expand
our work to large vocabulary translation tasks. The major difference of thisword and our work is the paralinguistic extraction method. In their work they
17

370

375

380

385

handle emphasis as a level between 0-1 that calculates similarity between an
HMM AM for emphasized speech and another HMM AM for normal speech.
Each word has one emphasis level feature and maps these emphasis levels
between input and target sequences. In their work, they need to annotate
a paralinguistic label for each type of paralinguistic information they wantto handle, and thus if they expand to other varieties of paralinguistic information (e.g. emotion or voice quality) they would need annotated training
data to do so. On the other hand, in our work we perform normal ASR toobtain alignments and extract observed features, and do not need to specify
specific linguistic labels.
State-of-the-art work on speech translation [4] translates input speech totarget words directly with sequential attentional model. In this work they
only focus linguistic features on target side and evaluate according to BLEU
score. There is also work that focuses on direct speech-to-text translation
using sequential attentional models [8, 23]. In this work, any paralinguisticfeatures that exist on the source side may be reflected in the lexical content
of the target translations, but paralinguistic information will not be reflectedin the target speech.
7. Conclusion

390

395

In this paper we proposed a generalized model to translate duration andpower information for speech-to-speech translation. Experimental results
showed proposed method can model input speech emphasis more effectively
than baseline methods. In future work we plan to expand beyond the digittranslation task in the current paper to a more general translation task using phrase-based or attention-based neural MT. The difficulty here is the
procurement of parallel corpora with similar paralinguistic information forlarge-vocabulary translation tasks. We are currently considering possibilities including simultaneous interpretation corpora and movie dubs. Anotheravenue for future work is to expand to other acoustic features such as F0,
which play an important part in other language pairs.
References

400[1] M. Abe, S. Nakamura, K. Shikano, and H. Kuwabara. Voice conversion
through vector quantization. In ICASSP-88., International Conferenceon Acoustics, Speech, and Signal Processing, volume 1, pages 655–658,
Apr 1988.
18

405[2] Pablo Daniel Agüero, Jordi Adell, and Antonio Bonafonte. Prosody generation for speech-to-speech translation. In IEEE International Conference on Acoustics Speech and Signal Processing, pages 557–560, 2006.
[3] Gopala Krishna Anumanchipalli, Luı́s C. Oliveira, and Alan W. Black.
Intent transfer in speech-to-speech machine translation. In 2012 IEEE
Spoken Language Technology Workshop (SLT), Miami, FL, USA, December 2-5, 2012, pages 153–158, 2012.

410

415

[4] Quoc Truong Do, Sakriani Sakti, and Satoshi Nakamura. Toward expressive speech translation: A unified sequence-to-sequence lstms approach for translating words and emphasis. In Interspeech 2017, 18th
Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pages 2640–2644, 2017.
[5] Quoc Truong Do, Sakriani Sakti, Graham Neubig, Tomoki Toda, andSatoshi Nakamura. Improving translation of emphasis with pause prediction in speech-to-speech translation systems. In 12th International
Workshop on Spoken Language Translation (IWSLT), Da Nang, Vietnam, December.

425[6] Quoc Truong Do, Shinnosuke Takamichi, Sakriani Sakti, Graham Neubig, Tomoki Toda, and Satoshi Nakamura. Preserving word-level emphasis in speech-to-speech translation using linear regression hsmms.
In INTERSPEECH 2015, 16th Annual Conference of the International
Speech Communication Association, Dresden, Germany, September 610, 2015, pages 3665–3669, 2015.

430[7] Markus Dreyer and Yuanzhe Dong. APRO: all-pairs ranking optimization for MT tuning. In NAACL HLT 2015, The 2015 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 June 5, 2015, pages 1018–1023, 2015.

420

435

[8] Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, andTrevor Cohn. An attentional model for speech translation without transcription. In NAACL HLT 2016, The 2016 Conference of the North
American Chapter of the Association for Computational Linguistics:
Human Language Technologies, San Diego California, USA, June 1217, 2016, pages 949–959, 2016.
19

[9] Jie Jiang, Zeeshan Ahmed, Julie Carson-Berndsen, Peter Cahill, andAndy Way. Phonetic representation-based speech translation.

440

445

450

455

460

[10] Takatomo Kano, Sakriani Sakti, Shinnosuke Takamichi, Graham Neubig, Tomoki Toda, and Satoshi Nakamura. A method for translation ofparalinguistic information. In International Workshop on Spoken Language Translation, pages 158–163.
[11] Takatomo Kano, Shinnosuke Takamichi, Sakriani Sakti, Graham Neubig, Tomoki Toda, and Satoshi Nakamura. Generalizing continuousspace translation of paralinguistic information. In INTERSPEECH
2013, 14th Annual Conference of the International Speech Communication Association, Lyon, France, August 25-29, 2013, pages 2614–2618,
2013.
[12] Philipp Koehn and Hieu Hoang. Factored translation models. In
EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on
Empirical Methods in Natural Language Processing and Computational
Natural Language Learning, June 28-30, 2007, Prague, Czech Republic,
pages 868–876, 2007.
[13] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch,
Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statisticalmachine translation. In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007,
Prague, Czech Republic, 2007.
[14] R. G. Leonard. A database for speaker-independent digit recognition. In
IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP ’84, San Diego, California, USA, March 19-21, 1984,
pages 328–331, 1984.

465[15] Shigeo Morishima and Satoshi Nakamura. Multi-modal translation system and its evaluation. In 4th IEEE International Conference on Multimodal Interfaces (ICMI 2002), 14-16 October 2002, Pittsburgh, PA,
USA, pages 241–246, 2002.

20

470

475

480

485

490

495

500

[16] Graham Neubig, Kevin Duh, Masaya Ogushi, Takatomo Kano, Tetsuo Kiso, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. The
NAIST machine translation system for IWSLT2012. In 2012 International Workshop on Spoken Language Translation, IWSLT 2012, Hong
Kong, December 6-7, 2012, pages 54–60, 2012.
[17] David Pearce and Hans-Günter Hirsch. The aurora experimental framework for the performance evaluation of speech recognition systems undernoisy conditions. In Sixth International Conference on Spoken Language
Processing, ICSLP 2000 / INTERSPEECH 2000, Beijing, China, October 16-20, 2000, pages 29–32, 2000.
[18] Vivek Kumar Rangarajan Sridhar, Srinivas Bangalore, and Shrikanth
Narayanan. Enriching machine-mediated speech-to-speech translation
using contextual information. Computer Speech & Language, 27(2):492–
508, 2013.
[19] Éva Székely, Ingmar Steiner, Zeeshan Ahmed, and Julie CarsonBerndsen. Facial expression-based affective speech translation. J. Multimodal User Interfaces, 8(1):87–96, 2014.
[20] Toshiyuki Takezawa, Tsuyoshi Morimoto, Yoshinori Sagisaka, NickCampbell, Hitoshi Iida, Fumiaki Sugaya, Akio Yokoo, and Seiichi
Yamamoto. A japanese-to-english speech translation system: ATRMATRIX. In The 5th International Conference on Spoken Language
Processing, Incorporating The 7th Australian International Speech Science and Technology Conference, Sydney Convention Centre, Sydney,
Australia, 30th November - 4th December 1998, 1998.
[21] Tomoki Toda, Alan W. Black, and Keiichi Tokuda. Voice conversion
based on maximum-likelihood estimation of spectral parameter trajectory. IEEE Trans. Audio, Speech & Language Processing, 15(8):2222–
2235, 2007.
[22] Wolfgang Wahlster. Robust translation of spontaneous speech: A multiengine approach. In Proceedings of the Seventeenth International Joint
Conference on Artificial Intelligence, IJCAI 2001, Seattle, Washington,
USA, August 4-10, 2001, pages 1484–1493, 2001.

21

[23] Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng
Chen. Sequence-to-sequence models can directly transcribe foreign
speech. CoRR, abs/1703.08581, 2017.

505[24] Heiga Zen, Keiichi Tokuda, and Alan W. Black. Statistical parametricspeech synthesis. Speech Communication, 51(11):1039–1064, 2009.
[25] J Zhang and Satoshi Nakamura. An efficient algorithm to search for aminimum sentence set for collecting speech database. In International
Congress of Phonetic Sciences, pages 3145–3148, 2003.

22