IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

1

Sequence-to-Sequence Models for
Emphasis Speech Translation
Quoc Truong Do, Student Member, IEEE, Sakriani Sakti, Member, IEEE, and Satoshi Nakamura, Member, IEEE

Abstract—Speech-to-speech translation (S2ST) systems arecapable of breaking language barriers in cross-lingual communication by translating speech across languages. Recent studies haveintroduced many improvements that allow existing S2ST systems
to handle not only linguistic meaning but also paralinguistic
information such as emphasis by proposing additional emphasis
estimation and translation components. However, the approach
used for emphasis translation is not optimal for sequence translation tasks and fails to easily handle the long-term dependencies
of words and emphasis levels. It also requires the quantizationof emphasis levels and treats them as discrete labels insteadof continuous values. Moreover, the whole translation pipeline
is fairly complex and slow because all components are trained
separately without joint optimization. In this paper, we maketwo contributions: (a) we propose an approach that can handle
continuous emphasis levels based on sequence-to-sequence models, and (b), we combine machine and emphasis translation intoa single model, which greatly simplifies the translation pipeline
and make it easier to perform joint optimization. Our results
on an emphasis translation task indicate that our translation
models outperform previous models by a large margin in bothobjective and subjective tests. Experiments on a joint translation
model also show that our models can perform joint translationof words and emphasis with one-word delays instead of fullsentence delays while preserving the translation performance ofboth tasks.
Index Terms—Emphasis estimation, emphasis translation,
speech-to-speech translation, joint optimization of words andemphasis.

I. I NTRODUCTIONPEECH is one of the world’s richest and most powerfulcommunication channels, allows speakers to express not
only the content that they want to convey but also paralinguistic information such as emotion and emphasis. Emphasis
is often used to distinguish between the focused and unfocusedparts of an utterance [1] and is particularly useful in misheard
situations where speakers need to repeat the most importantwords or phrases of sentences. In speech-to-speech translation
tasks, Tsiartas et al. [2] has conducted a study on multilingual speech corpora and argued that emphasis information
is a critical factor that contributes to the quality of speech-tospeech translation performance.
Many studies have developed and improved automatic S2ST
translation systems [3] that help translate the content ofspeech across languages. An S2ST system consists of 3 main
components: automatic speech recognition (ASR), machine

S

Q.T. Do is with Nara Institute of Science and Technology, Japan. E-mail:do.truong.dj3@is.naist.jp.
S. Sakti and S. Nakamura are with Nara Institute of Science and Technology
and RIKEN Center for Advanced Intelligence Project AIP, Japan. Emails:
ssakti@is.naist.jp, s-nakamura@is.naist.jp

atsui desu

It is hot
MT

ASR

ES

ET (CRF-based)

Q

0.12 0.2 0.87

TTS

0.9 0.1

0.1 0.3 0.9

Fig. 1. Existing emphasis speech translation model that consists of many separate components and dependencies and also requires emphasis quantization(Q) before translation.

MT

ASR

TTS

ET (hard-attentional)

ES

0.83 0.19

0.12 0.2 0.87

Fig. 2. Proposed hard-attention emphasis speech translation model that cantranslate continuous emphasis weights without quantization.
ASR

MT ET

TTS
0.72 0.19

ES
0.12 0.2 0.87

Fig. 3. Proposed joint model simplifies translation pipeline and jointly
translates words and emphasis with one-word delay.

translation (MT), and text-to-speech synthesis (TTS). However, since most S2ST systems cannot translate emphasis conveyed in the source language speech, communication throughtraditional S2ST systems is less engaging than natural speechcommunication.
The difficulty in handling emphasis is that it can be manifested by changing different types of acoustic features, suchas the duration, the power, or the F0 of the emphasizedwords [4]. The challenge in developing an S2ST system that
can accurately translate emphasis is that we must consider
these acoustic features of emphasis in three components:emphasis extraction, emphasis translation, and the synthesisof emphasized speech. Kano et al. [5,6] proposed approaches
to translate emphasis in a limited domain of 10 digits. Since
their approach models speech differently for each word in the
vocabulary, they cannot generalize to unseen words, and theyalso have difficulty modeling emphasis in large vocabularysystems. Anumanchipalli et al. [7], Aguero et al. [8], andTsiartas et al. [9] proposed approaches for mapping F0 intoa discrete set of units and translating them across languages.
However, other acoustic features such as duration and power,

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

were not taken into account.
The most recent work by Do et al. [10] attempts to translateemphasis in an open domain. Their basic idea is to represent
emphasis as a continuous real-numbered value (emphasis
level) that is estimated using all of the acoustic features.
Then they use the conditional random fields to translate theestimated emphasis levels to a target language. Using emphasis
levels makes the emphasis representation more intuitive andeasier to translate because emphasis translation can be viewed
as translating a sequence of numbers.
However, the following three problems remain: (1) Emphasis translation is based on conditional random fields (CRFs)
that require emphasis levels of quantization that are treatedas discrete labels instead of continuous values. This leads
to objective functions that cannot capture the “amount ofdifference” between emphasis levels. For examples, the levelsof 0.7 and 0.6 are quite close if they are considered realnumbered values, but they are completely different if we
treat them as discrete labels. (2) CRFs are not optimal forsequence translation tasks and cannot easily handle the longterm dependencies of words and emphasis levels. (3) Since
the entire translation pipeline is fairly complex and slow,
all downstream components have to wait for the complete
result of the upstream components, delaying full-sentencetranslations (Fig. 1). Moreover, all of the components aretrained separately and glued together to perform decoding.
Joint optimization cannot be directly applied because of thecomplexity of the translation pipeline.
In this paper, we make two contributions: (a) we propose
an approach that can handle continuous emphasis levels basedon sequence-to-sequence (seq-to-seq) models (Fig. 2). The
objective function is the mean square error that directly takes
into account the “amount of difference” of the emphasis levels.
(b) We combine both machine and emphasis translations intoa single joint translation model (Fig. 3) based on seq-to-seq
attention models that achieved the state-of-the-art performancein machine translation tasks [11,12]. As the result, the translation pipeline is greatly simplified and we can perform joint
optimization. We can also avoid one-to-one word alignments,
which are a required component of previous works. Thismechanism not only reduces the complexity but also speeds
up the decoding process1 .

2

A. Automatic Speech Recognition
The ASR component, which is the first element of the
S2ST translation pipeline, transcribes input speech signal x
into corresponding word sequence w. Input x is decomposed
into a sequence of feature frames o that only retain relevant
information for the ASR task. Word sequence w is thenpredicted to maximize the conditional probability,ŵ = argmax P (w|o).

For high accuracy, such speaker-dependent features as emotion and emphasis are either removed or normalized. As aresult, emphasis information is lost after the ASR component.
B. Statistical Machine Translation
The MT component translates a word sequence from asource language into the target language. Many methods canbe applied to the MT task, including phrase-based [15], treebased [16], and neural network [11] translation models.
In the S2ST system, the MT component is downstream of
the ASR component that takes ASR output w(s) and finds thehighest probability target language sentence w(t) :ŵ(t) = argmax P (w(t) |w(s) ).

1 Parts of this work were previously presented [13,14]. The current work
provides a more comprehensive and systematic description of our method anda deeper analysis of our experiments.

(2)

w(t)

Similar to the ASR component limitation, the MT is only
optimized to translate linguistic information and cannot handleemphasis information.
C. Text-to-speech Synthesis
Text-to-speech, which is the last component in the S2ST
system, synthesizes the target audio given the translated text
from the MT component. Many approaches are also used in
TTS such as DNN-based or HMM-based. While DNN-based
approaches are recently getting more attention because of their
good synthetic speech, HMM-based approaches provide much
more flexibility for handling emotion or emphasis information.
The general idea of the HMM-based TTS approach is that
output speech parameter vector sequence v is determined by
maximizing the likelihood function where the state sequence
consists of T states q = [q1 , · · · , qT ] and HMM model set Mv̂ = argmax P (W v|q, M),

II. CRF- BASED E MPHASIS S PEECH - TO -S PEECHT RANSLATIONIn this section, we describe the basic components of anemphasis speech translation (E-S2ST) system and the mostrecent state-of-the-art system based on CRF [10].
As described in the previous section, there are many works
on emphasis translation. Although they utilized different approaches, the translation pipeline is the same as illustrated inFig. 1. It combines 2 main sub-systems: an S2ST system thattranslates the linguistic meaning of speech with ASR, MT,
and TTS modules; and an emphasis translation system that
estimates (ES) and translates the emphasis information (ET).

(1)

w

(3)

v

where W is the weighting matrix for calculating the dynamic
features [17].
Unlike ASR and MT, the TTS component can be optimized
for synthesizing emphasis speech [1]. However, because theMT component’s output only contains text, the output soundcannot be emphasized in a way that reflects the originalemphasis of the source language.
D. Emphasis Estimation
To address ASR’s limitation, the ES component was proposed to estimate emphasis information using emphasis features including power, F0 , and duration. Do et al. [10] proposed an approach based on a linear-regression hidden semiMarkov model (LR-HSMM) [18] that models emphasis atIEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

the word-level (emphasis weight). The emphasis weight is areal-numbered value that represents the intensity of a word’semphasis. As illustrated in Fig. 1, since the ES system takeemphasis features as inputs and the ASR outputs word sequences, each word from the ASR output has a correspondingemphasis weight number.
The intuition of the emphasis weights is shown in Fig. 4.
Given a word sequence, 2 HSMM state sequences, normal
and emphasis, can be derived, both of which are respectively trained from normal and emphasized speech. Emphasis
weights interpolate the mean component of these 2 HSMM
sequences to construct an LR-HSMM sequence. Note that theemphasis weights are shared among all HSMM states thatbelong to one word.
In the model training stage, only normal and emphasis
HSMM parameters are optimized. Inn the emphasis estimation stage, emphasis weights λ are optimized using the EM
algorithm to maximize the HSMM likelihood as follows:λ̂ = argmax P (o|λ, M),(4)
λwhere o is an observation speech feature sequence and M is
the LR-HSMM parameters.
How

are
...

...

...

EmphasisHSMM

...

...

...

λ2

...

III. L IMITATIONS
A. CRF-based Emphasis Translation
Even though a CRF-based ET can preserve emphasis, itsmajor problem is that it must quantize continuous emphasis
levels into discrete labels. Although this mechanism increases
the ratio of the number of labels and their training samples,
the translation model is prone to make very bad predictions.
For instance, instead of predicting 0.9, it might predict 0.1.
Since those values are treated as separate discrete labels, itcannot capture the difference between 0.9 and 0.1.
Another problem with the CRF-based approach is that
although it model local dependencies well (by adding morefeature functions), it has difficulty handling long-term dependencies. One can use many feature functions to handle
this problem, but as they increase, more data are required.
And since emphasis translation requires parallel emphasized
speech, which is very hard to collect, this approach is notpractical.
B. Complex Translation Pipeline
To translate emphasis, the translation pipeline requires ES
and ET components in addition to the S2ST system, which is
now very complex: 5 components, and 6 internal dependencies(represents by black arrows) (Fig. 1). All the downstream
components have to wait for the upstream outputs, resulting inlarge translation delays. Moreover, each component uses very
different techniques, complicating joint training and decoding.

λ3Interpolation
LR-HSMM

where N is the number of training samples, K is the numberof feature functions f , and θk is the weight parameter offeature function k-th. Featurefunction fk combines word-level
(t)′
(t)′
(k)
emphasis bi-gram λn−1 , λn , and input feature xn .

you

NormalHSMM

λ1

3

...

...

Fig. 4. Continuous emphasis modeling with linear-regression HSMM.

IV. S EQUENCE - TO - SEQUENCE A PPROACHES FOR
E MPHASIS T RANSLATION
E. Emphasis Translation (ET)The ET component plays a similar role as the MT component. The difference is that instead of translating a sequence ofwords, it translates a sequence of estimated emphasis weights
described in II-D. As illustrated in Fig. 1, the ET component
takes 2 inputs, estimated source-language emphasis sequence
λ(s) and translated word sequence w(t) , and predicts targetlanguage emphasis sequence λ(t) .
Do et al. [10] proposed an emphasis translation approachbased on conditional random fields(CRFs)[19] that treats′′emphasis as ′discrete labels2 λ(s) and λ(t) . Target emphasis
weights λ(t) are then modeled by the following likelihood
function:
(K
)
N
X
Y
′
(t)′
(k)
exp
θk fk (λn−1 , λ(t)n , xn )′P (λ(t) |x) =

n=1

N
X Y

λ̃

(t)′ n=1

exp

k=1
(K
X

k=1

′
(t)′
θk fk (λ̃n−1 , λ̃n(t) , xn(k) )

),

(5)

2 Estimated continuous emphasis weights are quantized into sets of buckets(labels)

In this section, we describe our proposed approaches totackle the above limitations. We first propose a hard-attentional
seq-to-seq long short-term memory (LSTM) model that does
not require emphasis quantization while retaining the ablility
to capture the complex dependencies of emphasis and otherlinguistic information such as words and part-of-speech (PoS)
tags. Then we propose a new emphasis speech translation
pipeline that combines MT and ET into a new single model,
which eliminates complex dependencies and makes the entire
translation faster.
We chose the seq-to-seq based approach because it has
achieved impressive results for many tasks, such as speech
recognition [20,21] and machine translation (MT) [11]. Particularly, attentional-based seq-to-seq [22,23] achieved stateof-the-art performances for MT and ASR tasks and can model
long-term dependencies, overcoming the problems of local
dependencies in CRFs. In addition, models can be definedthat can simultaneously handle both continuous and discretevariables, as well as cost functions that take into account label
distances, for example, mean squared errors.
Moreover, since the state-of-the-art MT is based on seqto-seq approaches and ET also resembles a translation task,

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

better performance can probably be achieved utilizing seq-toseq for ET tasks. In addition, integrating MT and ET tasks
will become easier because similar techniques are applied forboth tasks.
A. Sequence-to-sequence LSTM Models

4

B. Hard-attentional Seq-to-seq Emphasis Translation
Our proposed hard-attentional model for emphasis translation is a modified version of the seq-to-seq model describedin the above section, based on an assumption that we have atarget language word sequence that was predicted from anexternal MT model and word alignments from an external
word alignment model.

LSTM [24] is a special kind of recurrent neural networkmodel that can capture long-term dependencies by special
units called memory blocks and also manages the information
going through it using forget, input, and output gates. Given
input vector xt at time t and hidden vector ht−1 and cell state
Ct−1 at time t − 1, the information flow can be described:• Calculate forget gate ft :ft = σ(Wf × [ht−1 , xt ] + bf ).
•

•

•

e t:
Calculate input gate it and estimate cell state C
it
e
Ct

(6)

σ(Wi × [ht−1 , xt ] + bi ),(7)=tanh(WC × [ht−1 , xt ] + bC ),

(8)

(9)

Calculate output vector ht :σ(Wv × [ht−1,xt ] + bv ),vt × tanh(Ct ).

=
=

(10)
(11)

Here W and b are the matrix and bias vectors of the neuralnetwork layers. The core component of an LSTM is cellstate Ct (Eq. (9)), which is controlled by forget gate ft
that is multiplied by the previous cell state values to decide
which history information it should forget, and input gate it ,which is multiplied to the estimated cell state to decide which
information is sent to the cell state.
An attention seq-to-seq LSTM model consists of an LSTM
encoder, which encodes the input information, an LSTM
decoder, which takes the encoded output to make a prediction,
and an attention layer, which calculates an attention vector.
The seq-to-seq translation model can be written as follows:(s)• Encode the input features to obtain hidden states h
:
(s)

hi
•

(s)

= enc(hi−1 , x).

(12)

(t)

Compute the attention vector aj and context vector cj :
(t)

(t)

(s)

aj = att(hj , hi ),

(13)
(t)

where j is the prediction time step, and hj is the decoder(t)hidden state. Given aj as weights, context vector cj is
computed as the weighted average over all source hidden
states h(s) .
• Predicts target labels yj ,P (yj |y<j , x)
e (t)
h
j

=
=

e (t) ),
sof tmax(Wt hj(t)tanh(Wc [cj ; hj ]).

LSTM cell

LSTM cellw (t1 ) - p(t1 )w (t2 ) - p(t2 )Decoderh1h2

Encoder

LSTM cell

LSTM cellx(s)1Update cell state Ct :vt
ht

λ(t2 )

(s)
(s)
w(s)
1 - p1 - λ 1=

e t,
Ct = ft × Ct−1 + it × C

λ (t1 )

0

(14)
(15)

x(s)
2

(s)
(s)
w (s)
2 - p2 - λ 2

Fig. 5. Unfolded hard-attentional encoder-decoder LSTM model for translating emphasis sequence λ(e) into target output sequence o(f ) . It considers(e,f )
many linguistic features including word sequence wi
and part of speech(e,f )
sequence pi
from both source and target languages.

The entire encoder-decoder process can be written as afunction of input features:
λ(t) = f (x(s) ),
(t)

(16)
(s)

where λ is the target output emphasis sequence, x
isthe sequence of the source-language input features includingwords w(s) , PoS p(s) , and emphasis weights λ(s) . A previous
work [10] reported that both words and PoS tags play a crucialrole in good translation models.
1) The encoder: As illustrated in Fig. 5, the encoder is a(e)standard LSTM model that takes input vector xi , which con(e)(e)
sists of words (wi ), part-of-speech tags (pi ), and emphasis(e)levels (λi ), and encodes them into a single vector that issuitable for predicting emphasis levels.
The input PoS tags are converted into one-hot vectors whose
size equals the PoS vocabulary size. Word embeddings [25] areapplied to map words onto vectors that capture the similarity
between words. All these input features are concatenated intoa single vector and fed to the encoder.
The encoder is pre-trained by appending a linear neuralnet layer on top of it with an output size of 1 to predict theemphasis level that is fed into the input layer, similar to an
auto-encoder model [26] (Fig. 6 (a)). We want output hidden
layer h to represent the features that are the most useful topredict the emphasis levels (called emphasis representations).
2) The decoder: The decoder is also a standard LSTM
model, and the input layer contains both linguistic information(words, PoS) and vector representations calculated by theencoder, based on a novel hard-attentional model.
The name hard-attentional reflects how the decoder calculates the emphasis representation vectors used as input.
The example in Fig. 5 demonstrates this mechanism. Assume
(s)
(t)
(s)
(t)
that word pairs w1 -w2 and w3 -w1 are aligned based on(t)word alignments. To generate output λ2 , linguistic features

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

(t)

(t)

5

(t)

w2 and p2 , and previous output λ1 , the decoder takes(s)
encoded h1 from the encoder output, because word pair w1 (t)w2 is aligned. For unaligned words, we use zero vectors asemphasis representation vectors.
We propose 2 decoders as follows:• LSTM_emph: The model directly predicts target emphasis sequence λ(t) .
• LSTM_diff: The model’s output is treated as the difference from the input emphasis level. The target emphasis(t)level of the j-th word is calculated by, λj = f (x(s) ) +(s)(s)
λi , where the model gets “attention” from word wi .
The intuition behind LSTM_diff is an intention to put stronger
weight on the corresponding source-language emphasis when
predicting target emphasis.
λ (e)

λ

(f )

W e , be

(s)

hi

(s)

hi

w -P

(b)

(f )

(s)

(s)

= [enc(wi ), ei ](t)e (t) , w(t) ]).
ei = W e ([hiiDecoder

(a)

(17)

(18)

The SkipEnc idea is that if emphasis weights negatively
affect machine translation, adding them after the encoder
might weaken the effect.
2) Decoder with emphasis weights: As illustrated in Fig. 7,
the decoder has two components. A word prediction layer
follows the standard NMT, and emphasis prediction layer W ethat takes input is the combined vector of the predicted word
and the decoder hidden activation as follows:hw (e) - P(e) - λ(e)(s)

By doing this, we ensure that the emphasis weights are also
encoded with words. However, since the effect of emphasis on
MT remains unknown, we need to explore alternative ways toincorporate emphasis into the encoder to analyze this effect.
Therefore, we propose adding emphasis after encoding words(SkipEnc) as follows:h

Encoder(s)= enc([wi , ei ]).

(f )

(19)
w (t2 )

(t )

e2
+

Fig. 6. Training procedure for the hard-attentional model.

c1a1e(ids)(a )C. Joint Translation Model
Even though the hard-attention seq-to-seq model solves
the problems of the CRF-based approach, which requires
emphasis quantization and suffers from the long-term dependency problem described in Section III-A, the problem witha complex translation pipeline remains. In this section, we
propose a joint translation framework based on an attentional
NMT that simultaneously combines MT and ET to translatewords and emphasis. Our approach is also based on seq-to-seq
approaches, as is the hard-attention approach. The difference isthat we do not require external MT or word alignment models
anymore. All components are combined into a single joint
translation model, allowing us to perform joint optimizationand inference.
When integrating emphasis with word translation, the majordifficulty is that the amount of text data usually overwhelms
the amount of emphasis data, because the latter are derived
from parallel emphasized speech that is much harder to collectthan parallel text data, which can be massively collected bycrawling websites [27].
We define the joint translation model as follows. Given asource language word and an emphasis sequence are denoted
as W (s) and e(s) , respectively. The model predicts one target
word w(t) at a time followed by a prediction of its emphasis
weight e(t) . Next, we detail how the encoder and decoder
handle both words and emphasis weights.
1) Encoder with emphasis weights: One way to embedemphasis weights into the encoder is to concatenate them with(s) (s)word representation to form input vector [wi , ei ] of theencoder (Emp-Enc) and compute the hidden unit:

1

(s)
(s)
w (s)
w (s)
1, e1
2, e2

(s)
w (s)
3, e3




w (t1 )

residualw (t2 )

Fig. 7. Joint word-emphasis translation framework with word dependencies
and residual connection.

However, as described above, the lack of emphasis datacompared with the text data might saturate the effect of thesource emphasis when going through many hidden layers. To
overcome this problem, we utilize a residual connection in
the way that the source emphasis weight is also used when
predicting target emphasis weights (Fig. 7),(t)e (t) , w(t) ]) + e(s) ,ei = W e ([hiiid(ai )

(20)

where function id(ai ) returns the index of the largest valueof weighted vector ai that indicates the source aligned word.
3) Training procedure: To train our model, we utilize two
objective functions, cross entropy (CE) for word prediction
and mean square error (MSE) for emphasis prediction, because
the CE function greatly outperforms MSE with discrete labels,
which is the case for word prediction. Since emphasis weights
are continuous, the CE function cannot be utilized as theobjective function for emphasis prediction.
The training algorithm is a standard back propagation
through time (BPTT) scheme in which the errors from themachine and emphasis translations are sequentially back–propagated. Note that the errors are not joint because theirscales are different.

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

V. E XPERIMENTS
In our experiment, we first evaluated the effectiveness ofusing a continuous emphasis level in a translation model andthe ability to handle the long-term dependencies of our proposed hard-attentional seq-to-seq model against the previous
CRF-based approach (Section V-B).
Regarding the evaluation of the joint translation model,
since our proposed model is our first attempt, to the best of
our knowledge, for integrating emphasis and word features ina single model, analysis must be conducted on the effect ofemphasis on the standard MT translation model (Section V-D).
This critical step not only shows the effect of emphasis as afeature, but also provides vital cues for optimally combining
ET and MT.
To further reduce the complexity of the input features andthe network structure, we also evaluated the effect of PoS
tags on the ET and MT models (Section V-E). Do et al. [10]argued that PoS tags are crucial features that can boost the ET
performance by a 4% F -measure. However, it creates anotherdependency for the translation model. On the other hand, theseq-to-seq translation model is capable not only of learning
how to translate but also learning the semantic meaning ofwords, and since the semantic meaning are closely related tosyntactic meaning (PoS tags) [28], we expect that it can avoid
the need of PoS tag features.
Finally, based on the analysis result, we conduct experiments with the joint translation model and compared both the
hard-attention and CRF-based approaches (Section V-F).
A. Experimental setup
1) Corpus: The corpus consists of emphasis and machinetranslation data. The former contain 966 parallel English andJapanese utterances [29]. In each language, at least one of thecontent words in the sentence is emphasized, and the numberof emphasized words is identical between languages. The
number of speakers is 8, including 3 native English (En{1,2,3} )and 5 native Japanese (Ja{1,2,3,4,5} ) speakers.
To create training and testing data for our emphasis translation evaluation, we divided 966 utterances of each speaker into
2 sets of 866 and 100 samples such that the same sentences areused for all speakers. We then paired the 866 utterances of eachEnglish speaker with those of all 5 Japanese speakers, resulting
in 4330 (866 ∗ 5) training, and 100 testing samples for eachEnglish speaker. The testing data consist of 157 emphasizedwords, in which 30 exist in the training data and 127 do not.
Regarding to the machine translation data, we utilized 2 sets,
the BTEC and BTEC+TED corpora, which contain ~450k and~670k parallel sentences, respectively. We created 2 training
MT datasets to evaluate the effectiveness of the emphasis
information on the MT task with more varieties of testing
conditions.
2) Emphasis translation procedure & measurement: In thispaper, to evaluate the performance of the emphasis translationin isolation, we assumed that the MT system produces 100%
correct translation outputs. Word alignments
To measure the emphasis translation accuracy, we first
performed emphasis translation to derive the target emphasis sequences and then measured its accuracy in the target

6

language both objectively or subjectively (Fig. 8). In theobjective evaluation, the target emphasis values are classified
as “emphasized” or “not emphasized” using a threshold of
0.53 and compared them with true values. In the subjective
evaluation, we first synthesized the audio from the translatedemphasis sequences, and gave the output audio to 7 Japanesenative listeners to predict the emphasized words4 . In both
evaluations, we calculated the F -measure, which ranged from0 to 100 representing how accurately the system preserved
emphasis in the target language.
Subjective evaluation

Objective evaluation
It

is

HOT

0.2

0.1

0.8

Emphasis translation

Speech synthesis

atsui desu
0.7 0.1
Classification

Human classification

Predict: ATSUI desu

Predict: atsui DESU

F-measure
Score: 1

Score: 0Label: ATSUI desu
(From the corpus)

Fig. 8. Example of the emphasis translation procedure and measurement
methods.

3) CRFs: We retained the identical configuration of the
CRF models as in previous work [30]. The word-level emphasis was quantized to the closest {0, 0.3, 0.6, 0.9}. The
input features are words, PoS tags, and PoS contexts in thetarget language side. The model directly predicts the target sideemphasis sequence. This setting achieved the best performancecompared to other features combinations.
4) Hard-attentional Seq-to-seq model:• The encoder: The encoder input consists of words,
PoS tags, and emphasis levels. The input layer has 138
dimensions including 100 word embeddings, 37 one-hot
PoS tags, and the emphasis levels. The hidden layer has
100 dimensions.
• The decoder: The input gate consists of 100 wordembedding dimensions and 17 one-hot PoS dimensions.
The attentional vector taken from the encoder was added
to the input gate’s output. The input words and PoS
are also respectively converted into word-embedding andone-hot vectors.
The word embeddings for both the encoder and decoder
were pre-trained using the BTEC travel conversation corpus[31] using word2vec toolkit [25].
5) Joint translation model: Our encoder and decoder models have 1 layer (unless stated otherwise), 512 cells, and 512dimensional word embeddings. We trained for a maximum
3 This has been reported in the previous work [30] as having the best
performance to classify emphasized and normal words.
4 There is no constraint on how emphasized words are expressed, it is up
to the listeners to make a binary decision on whether a word is emphasized.

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

of 20 epochs using the RMSprop algorithm [20]. Emphasis
prediction layer W e was frozen when trained with fakeemphasis data to avoid learning from unrealistic emphasis
weights.
When trained with text data, the learning rate was set to 1e−
4 and 5e−5 when trained with emphasis data. We employed anearly stop learning rate schedule and reduced the learning bya factor of 2 whenever loss increased on the development set
and stopped the training when the learning rate fell below 1e5. Our mini-batches were 128 and 10 for the word translationand the emphasis translation task, respectively. The batches
were shuffled before every training epoch.
B. Hard-attentional models: objective evaluation
We performed a preliminary experiment using the same
corpus as in a previous work [30] with 916 training samples
and 50 testing samples. The results showed that our proposed
method achieved a 92.6% F -measure, which exceeds theprevious work by 1%. Although the dataset was too small
to conclude that the proposed method is better than CRFs
by such a small margin, it demonstrates that the proposed
method performs comparably with the previous work on thesame corpus. To make the result more reliable, we conductedlarger scale experiments with the dataset described in the
Section V-A1.
Average:
LSTM_diff: 83.85%
LSTM_emph : 84.54%
CRFs: 80.73%92
90

88.46 88.82

88

F-measure

86

84.75

84

82.85 82.51

82.24
82
80.13
80

79.17
78.23

78
76
74

En1

En2

En3

Fig. 9. Objective emphasis prediction of hard-attentional enc-dec with
LSTM_diff and LSTM_emph architectures.

Fig. 9 shows the objective F -measure for emphasis prediction on this larger amount of data. In all 3 test sets andon average, the proposed methods outperformed the CRFs.
According to the bootstrap resampling significance test [32],both results are significant at the p < 0.01 level. On the other
hand, the difference between LSTM_diff and LSTM_emph was
not significant, demonstrating that the LSTM model can learnemphasis level differences between aligned words without
explicitly defining them in the equations.
Furthermore, we scrutinized the advantage of the proposedmodel with respect to using continuous variables. If they areuseful, we expect that the emphasis values in the middle of
the range will be modeled better by the proposed method. To
test this hypothesis, we split the input emphasis levels into
3 sets based on the emphasis level of the word: < 0.3, 0.30.6, > 0.6. Then we calculated F -measure for the CRFs and

7

LSTM_emph on individual sets5 . The result in Table I indicatesthat both systems have equivalent performance when a word isconsidered normal or emphasized (emphasis levels below 0.3
or over 0.6), but when the emphasis levels fall between 0.30.6, LSTM_emph outperformed the CRFs. This demonstrates
the limitation of the CRFs, which require emphasis level
quantization to handle continuous variables, but LSTMs donot.
TABLE IF - MEASURE FOR CRF AND LSTM_ EMPH EMPHASIS TRANSLATION ON
DIFFERENT INPUT EMPHASIS LEVELS .
CRF
88.05

<0.3
LSTM
87.69

0.3-0.6
CRF
LSTM
70.85
81.41

CRF
92.53

>0.6
LSTM
92.75

C. Hard-attentional models: subjective evaluation
Finally, we performed a subjective evaluation to verify
whether human listeners can perceive the same improvement
between CRFs and LSTM_emph as in the objective evaluation.
We used the “En1” test set for this evaluation.
We obtained a result of 83.0% for LSTM_emph and 81.0%
for CRFs indicating that humans perceived a slightly smaller
improvement compared to the objective result. Moreover, theCRF system’s performance dropped with a smaller margin(3.70%) than the proposed method (5.82%). The reason is
because in the LSTM_emph approach, 268 emphasized words
were recognized correctly in the objective evaluation, but 14of them having emphasis levels fall between 0.5-0.8 are misrecognized by human listeners while this does not happen in
the CRF approach since these emphasis levels are just slightly
higher than the threshold, leading to slightly emphasizedsynthetic speech that is hard to perceive by human listeners.
In the CRF approach, the emphasis levels are quantized into
buckets of {0, 0.3, 0.6, 0.9, . . . }, so when a word is considered
as emphasized (larger than the threshold 0.5), the distance tothe threshold is usually large.
D. Effect of using emphasis as additional features on standard
NMT systems
Even though previous works translated emphasis weights
separately from NMT, no analysis has addressed whether
emphasis weights in NMT have a positive or negative effect.
Such analysis, however, is important before integrating emphasis translation into NMTs. To address this oversight, we
explored the effect of emphasis as an input feature on machinetranslation performance.
We kept the same decoder structure like standard NMT
systems so that no emphasis prediction was performed andevaluated two encoders with emphasis weights added in different positions as described in Section IV-C1. The baselineis the standard NMT system without emphasis weights (Std.
NMT). Fig. 10 shows the result of the cross entropy loss of theword prediction performance on the training and development
5 Because the accuracies of LSTM_diff and LSTM_emph are similar, below
we only show the results of CRFs and LSTM_emph.

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

8

3

6
5

2

Std. NMTEmp-SkipEnc
Emp-Enc
Std. NMTEmp-SkipEnc
Emp-Enc

1.5
14
Perplexity

Cross entropy loss

2.5

3
w/ posw/t pos

2
1

0.5
0
1

0

2

3

4

5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

6

7

8

9 10 11 12 13 14 15 16 17

Training Epoch

Training epoch

sets. The loss is higher in both approaches (SkipEnc and
Emp-Enc) than in Std. NMT, indicating that emphasis did not
improve the NMT performance. We hypothesize that such anegative effect is due to the fact that emphasis weights areparalinguistic, but NMT is translating linguistic information.
Only using emphasis weights as an additional feature withouttranslating is insufficient for the model to learn anything usefulfrom emphasis.
Although the NMT performance was degraded when using
the emphasis weight features, SkipEnc has a minimal effectcompared with Emp-Enc. This is because in SkipEnc, theencoder avoids excessive influence from the negative effectof the faked emphasis weights; therefore, we can preserve theperformance of the standard NMT. The rest of our experiments
used the SkipEnc model.
E. Joint translation models: Effect of PoS tags on ET and MT
modelsFigure 11 shows the performance of the ET model using the
EmpEnc joint translation approach with and without the PoS
tag feature. With PoS tags, the model converges faster andprovides better performance in the first 10 iterations. But both
systems eventually converge to a similar point when we train
them for 15 iterations. We also observed the same tendency
in the MT task (Fig. 12). The result indicates that PoS tags
still help the translation model, but if we train it on a sufficientamount of iterations, such help is minimalized. We hypothesizethat this is because the model can learn semantic meaning ofa word that is similar to what the PoS tags represent.

Fig. 12. MT performance in joint translation models with/without PoS tags.

F. Joint translation models: Emphasis translation performance
From the result of the above sections, we conducted thefollowing experiments using SkipEnc architecture without PoS
tag features and completely trained the model for both emphasis and word prediction. Fig. 13 shows the F -measure,the precision, and the recall for emphasis prediction using the
SkipEnc encoder with baseline and residual decoders.
Looking at the F -measure, the residual decoder outperformed the baseline decoder by a 2.7% F -measure. The baseline decoder’s precision, however, is higher than the residual
one, indicating that the residual connection mistakenly predicts
more high emphasis weights for normal words. Similarly, thehigh score for the residual decoder’s recall indicates that itpreserves more emphasized words than the baseline system.
The contrastive precision and recall performance of the twosystems indicates that better performance is gained by combining them. In the next section, we describe our combination
technique and compare its result with previous works.
90
83.59

85

F-measureFig. 10. Effect of emphasis on standard NMT systems: Solid and dash lines
denote MT performance on development and the training sets, respectively.

80
72.96

75
70

69.04

66.74

65.58

65
60

F-scorePrecision
Recall

55.56

55
50

SkipEnc-Base

SkipEnc-Residual

Fig. 13. Emphasis translation performance in joint translation model

0.8

G. Joint translation models: model combination for emphasis
translation

0.7
0.6
RMSE

0.5
0.4

w/ PoS
w/t PoS

0.3
0.2
0.1
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Training EpochFig. 11. ET performance in joint translation models on a development set
with/without PoS tags.

The model combination works as follows. First, we performed emphasis translation on the development set andcalculated the precision and recall scores. Then, for content
words, we selected the emphasis weights predicted from thesystem with higher recall, and for the non-content words, weselected emphasis weights with lower recall.
We also performed emphasis translation using previous approaches based on conditional random fields (CRFs) [30] andLSTM hard-attention models [13]. The input features for theseapproaches are words and emphasis weights that resemble theIEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

proposed approach. The result is shown in Fig. 14. Comparedwith CRFs, our proposed approaches perform bettered witha ~5% F -measure and have a closed performance with the
LSTM hard-attention approach with a ~2% lower F -measure.
The result matches our expectation because both the CRFs
and LSTM hard-attention approaches use ground-truth oneto-one word alignments and have independent words andemphasis translation models. On the other hand, our proposed
approaches do not require word alignment models and cantranslate words and emphasis twice as fast as hard-attention
models.
85

80.77

F-measure

TABLE II
M ACHINE TRANSLATION PERFORMANCE (BLEU SCORE ) IN JOINT
TRANSLATION MODELS . VARIOUS DEPTHS OF HIDDEN LAYERS DENOTED
AS d(1,2,3) WERE EVALUATED .
SystemBaseline (d1)
SkipEnc-Base (d1)
SkipEnc-Residual (d1)Baseline (d2)
SkipEnc-Base (d2)
SkipEnc-Residual (d2)Baseline (d3)
SkipEnc-Base (d3)
SkipEnc-Residual (d3)

BTEC
27.67
27.25
27.19
27.44
27.70
27.72
23.68
25.41
26.36

BTEC + TED
24.20
23.16
24.96
25.43
25.14
21.44
22.24
23.22

78.38

80
75

9

73.8

70
65
60
55
50
CRF

hard-attention

Combination

Arch.

Separated

Separated

Joint

Align.

Required

Required

No

Delay

Full sentence

Full sentence

One word

Fig. 14. Comparison of emphasis translation performance of proposed andprevious approaches. Graph also shows differences in terms of translationarchitecture (Arch.), word alignment requirement (Align.), and the translation
delay (Delay).

H. Joint translation models: machine translation performance
Next, we evaluated the machine translation performancein joint translation models with 2 dataset, BTEC and
BTEC+TED. We are interested to see how the systems behave
when the number of parameters (model’s depth) increase. The
results are shown in Table II. The baseline system is thestandard NMT without the emphasis weights used in both theencoder and decoder.
First, we can see that the performance on the BTEC +TED set is lower than the BTEC set. It is because the TED
data covers a much more open domain, therefore harder totranslate, than the BTEC (only travel) domain. Second, whenthe model’s depth is 1 and 2, the performance difference of the
proposed approaches and the baseline is negligible for both test
datasets, indicating that optimizing the model with emphasis
weights can compensate for the negative effect of emphasisfound in Section V-D.
With a hidden layer depth of 3, all of the models seemover-fitted with the training samples, resulting in a loss of
performance. However, interestingly, the proposed approaches
have smaller performance drops. Specifically, the SkipEncResidual approach only dropped ~1-2% of BLEU, and thebaseline system without emphasis weights dropped ~3-4%of BLEU. We hypothesize that emphasis weights work as
regulation parameters that help preventing over-fitting.
VI. C ONCLUSIONSIn this paper, we proposed methods to accurately translateemphasis, and reduce translation complexity. Unlike previous

work where emphasis is considered to be discrete labels andhas difficulty handling long-term dependencies, our proposed
hard-attention seq-to-seq model can solve both problems ina single model by utilizing the LSTM-based encoder-decoderthat can capture long-term dependencies and handle continuous emphasis in its objective function. The evaluation onemphasis translation task demonstrates that our model cantranslate emphasis significantly better than previous work.
With regards to effect emphasis and PoS tags on a machinetranslation task, we discovered that emphasis does not help
standard MT systems if it is simply used as an additionalfeature. Experiments with PoS tags also showed that it helps
the model converge faster, but it does not help improve
the accuracy if the model is well-trained. Another important
outcome of our result is that our proposed model can learn
good features from words and emphasis without PoS tag
dependencies.
Our work on the joint translation of words and emphasis
demonstrated that our proposed joint translation model canaccurately translate emphasis and words with one-word delay,
but the previous work requires a full-sentence delay. The
model significantly reduced the complexity by removing wordalignments and PoS tag features. We also found that emphasis
can help MT performance prevent over-fitting.
However, some limitations remain. First, although the complexity is already reduced, we still require an emphasis estimation (ES) that works independently with the ASR component. Although this architecture allows us to adopt any ASR
technique without interfering with the emphasis estimation
component, it creates a delay during which ET and MT models
have to wait for the ASR and ES output.
Future work will integrate ES and ASR to completely
remove any dependencies introduced by adding emphasis
translation to standard S2ST systems. Joint training the wholesystem is another very interesting topic. Thanks to the seq-toseq model, we can apply it to all components to seamlessly
integrate them into a joint translation model. In addition, recent
works on speech recognition have shown that TDNN is comparable (or even better in certain cases) with LSTM. Integratingthese models into emphasis estimation and translation willfurther reduce the model complexity and potentially speed up
the translation pipeline.

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

VII. ACKNOWLEDGEMENTS
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237.
R EFERENCES[1] K. Yu, F. Mairesse, and S. Young, “Word-level emphasis modelling in
HMM-based speech synthesis,” in Processing of ICASSP, March 2010,
pp. 4238–4241.
[2] A. Tsiartas, P. G. Georgiou, and S. S. Narayanan, “A study on the effectof prosodic emphasis transfer on overall speech translation quality,” inProceedings of ICASSP, 2013.
[3] S. Nakamura, “Overcoming the language barrier with speech translation
technology,” Science & Technology Trends - Quarterly Review No.31,
April 2009.
[4] E. Fudge, English Word-stress. Routledge, 2015.
[5] T. Kano, S. Sakti, S. Takamichi, G. Neubig, T. Toda, and S. Nakamura,“A method for translation of paralinguistic information,” in Proceedings of the International Workshop on Spoken Language Translation(IWSLT), 2012.
[6] T. Kano, S. Takamichi, S. Sakti, G. Neubig, T. Toda, and S. Nakamura,“Generalizing continuous-space translation of paralinguistic information.” in Proceedings of INTERSPEECH, 2013, pp. 2614–2618.
[7] G. K. Anumanchipalli, L. C. Oliveira, and A. W. Black, “Intent transfer
in speech-to-speech machine translation,” in Proceedings of SLT, Dec
2012, pp. 153–158.
[8] P. D. Aguero, J. Adell, and A. Bonafonte, “Prosody generation forspeech-to-speech translation,” in Proceedings of ICASSP, vol. 1, 2006.
[9] A. Tsiartas, P. Georgiou, and S. S. Narayanan, “Toward transfer of acoustic cues of emphasis across languages,” in Proceedings of InterSpeech,
2013.
[10] Q. T. Do, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Preserving
word-level emphasis in speech-to-speech translation,” IEEE, vol. 25,
no. 3, pp. 544–556, March 2017.
[11] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learningwith neural networks,” in Proceedings of NIPS, 2014, pp. 3104–3112.
[12] M. T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser, “Multitask sequence to sequence learning,” in Proceedings of ICLR, 2016.
[13] Q. T. Do, S. Sakti, G. Neubig, and S. Nakamura, “Transferring emphasisin speech translation using hard-attentional neural network models,” inProceedings of Interspeech, September 2016.
[14] Q. T. Do, S. Sakti, and S. Nakamura, “Toward expressive speech translation: A unified sequence-to-sequence LSTMs approach for translatingwords and emphasis,” in Proceedings of Interspeech, 2017.
[15] P. Koehn, F. J. Och, and D. Marcu, “Statistical phrase-based translation,”in NAACL, 2003, pp. 48–54.
[16] Y. Liu, Q. Liu, and S. Lin, “Tree-to-string alignment template forstatistical machine translation,” in ACL, 2006, pp. 609–616.
[17] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura,“Speech parameter generation algorithms for HMM-based speech synthesis,” in Proceedings of ICASSP, vol. 3, 2000, pp. 1315–1318 vol.3.
[18] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi, “A style control
technique for HMM-based expressive speech synthesis,” IEICE, vol.
E90-D, no. 9, pp. 1406–1413, Sep. 2007.
[19] J. D. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditional random
fields: Probabilistic models for segmenting and labeling sequence data,”
in Proceedings of ICML, 2001, pp. 282–289.
[20] A. Graves, “Generating sequences with recurrent neural networks,”
CoRR, vol. abs/1308.0850, 2013.
[21] A. Graves, A. Mohamed, and G. Hinton, “Speech recognition with deep
recurrent neural networks,” in Proceedings of ICASSP. IEEE, 2013,
pp. 6645–6649.
[22] M. T. Luong, H. Pham, and C. Manning, “Effective approaches
to attention-based neural machine translation,” arXiv preprint
arXiv:1508.04025, 2015.
[23] K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and
Y. Bengio, “Show, attend and tell: Neural image caption generation withvisual attention,” arXiv preprint arXiv:1502.03044, 2015.
[24] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neuralcomputation, vol. 9, no. 8, pp. 1735–1780, 1997.
[25] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation ofword representations in vector space,” arXiv preprint arXiv:1301.3781,
2013.
[26] P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol, “Extracting andcomposing robust features with denoising autoencoders,” in Proceedingsof ICML, 2008, pp. 1096–1103.

10[27] J. Tiedemann, “Parallel data, tools and interfaces in OPUS,” in Proceedings of LREC, 2012.
[28] D. Marcheggiani and I. Titov, “Encoding sentences with graph convolutional networks for semantic role labeling,” CoRR, vol. abs/1703.04826,
2017.
[29] D. Q. Truong, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Collection and analysis of a Japanese-English emphasized speech corpus,” inProceedings of Oriental COCOSDA, September 2014.
[30] Q. T. Do, S. Takamichi, S. Sakti, G. Neubig, T. Toda, and S. Nakamura,“Preserving word-level emphasis in speech-to-speech translation using
linear regression HSMMs,” in INTERSPEECH, 2015.
[31] G. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto, “Creating corpora
for speech-to-speech translation,” in Proceedings of EUROSPEECH,
2003, pp. 381–384.
[32] P. Koehn, “Statistical significance tests for machine translation evaluation.” in Proceedings of EMNLP, 2004, pp. 388–395.

Quoc Truong Do received his B.E. from Universityof Engineering and Technology, Hanoi, Vietnam, in
2013, and his M.S. from the Graduate School ofInformation Science, NAIST, Nara, Japan in 2015.
He is currently in the doctoral course at NAIST,Japan. He interested in speech and natural language
processing, with a focus on speech recognition, andspeech translation. He is a student member of ISCA,
and ASJ.

Sakriani Sakti received her B.E. degree in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In 2000, she received
DAAD-Siemens Program Asia 21st Century Award
to study in Communication Technology, University
of Ulm, Germany, and received her MSc degreein 2002. During her thesis work, she worked with
Speech Understanding Department, DaimlerChrysler
Research Center, Ulm, Germany. Between 20032009, she worked as a researcher at ATR SLC Labs,
Japan, and during 2006-2011, she worked as anexpert researcher at NICT SLC Groups, Japan. While working with ATRNICT, Japan, she continued her study (2005-2008) with Dialog Systems Group
University of Ulm, Germany, and received her PhD degree in 2008. She actively involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003-2007), A-STAR and U-STAR (2006-2011). In 2009-2011, she
served as a visiting professor of Computer Science Department, Universityof Indonesia (UI), Indonesia. From 2011, she has been an assistant professor
at the Augmented Human Communication Laboratory, NAIST, Japan. She
served also as a visiting scientific researcher of INRIA Paris-Rocquencourt,
France, in 2015-2016, under “JSPS Strategic Young Researcher Overseas
Visits Program for Accelerating Brain Circulation”. She is a member of JNS,
SFN, ASJ, ISCA, IEICE and IEEE. She is now also a board member of
SLTU (Spoken Language Technologies for Under-resourced languages) anda committee member of SIG ELRA-LRL (Low Resourced Languages). Her
research interests include statistical pattern recognition, speech recognition,spoken language translation, cognitive communication, and graphical modeling framework.

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

Satoshi Nakamura is Professor of Graduate Schoolof Information Science, Nara Institute of Science andTechnology, Japan, Honorarprofessor of Karlsruhe
Institute of Technology, Germany, and ATR Fellow.
He received his B.S. from Kyoto Institute of Technology in 1981 and Ph.D. from Kyoto University
in 1992. He was Associate Professor of Graduate
School of Information Science at Nara Instituteof Science and Technology in 1994-2000. He was
Director of ATR Spoken Language Communication
Research Laboratories in 2000-2008 and Vice president of ATR in 2007-2008. He was Director General of Keihanna ResearchLaboratories and the Executive Director of Knowledge Creating Communication Research Center, National Institute of Information and Communications
Technology, Japan in 2009-2010. He is currently Director of Augmented
Human Communication laboratory and a full professor of Graduate Schoolof Information Science at Nara Institute of Science and Technology. He isinterested in modeling and systems of speech-to-speech translation and speech
recognition. He is one of the leaders of speech-to-speech translation researchand has been serving for various speech-to-speech translation research projects
in the world including C-STAR, IWSLT and A-STAR. He received Yamashita
Research Award, Kiyasu Award from the Information Processing Society ofJapan, Telecom System Award, AAMT Nagao Award, Docomo Mobile Science Award in 2007, ASJ Award for Distinguished Achievements in Acoustics.
He received the Commendation for Science and Technology by the Minister of
Education, Science and Technology, and the Commendation for Science andTechnology by the Minister of Internal Affair and Communications. He also
received LREC Antonio Zampoli Award 2012. He has been Elected Board
Member of International Speech Communication Association, ISCA, sinceJune 2011, IEEE Signal Processing Magazine Editorial Board Member sinceApril 2012, IEEE SPS Speech and Language Technical Committee Member
since 2013, and IEEE Fellow since 2016.

11