IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

1

End-to-end Speech Translation withTranscoding by Multi-task Learning
for Distant Language Pairs
Takatomo Kano, Student Member, IEEE, Sakriani Sakti, Member, IEEE, and Satoshi Nakamura, Fellow, IEEE

Abstract—Directly translating spoken utterances from a source
language to a target language is challenging because it requires
a fundamental transformation in both linguistic and para/nonlinguistic features. Traditional speech-to-speech translation approaches concatenate automatic speech recognition (ASR), textto-text machine translation (MT), and text-to-speech synthesizer(TTS) by text information. The current state-of-the-art models
for ASR, MT, and TTS have mainly been built using deep neuralnetworks, in particular, an attention-based encoder-decoder neural network with an attention mechanism. Recently, several works
have constructed end-to-end direct speech-to-text translation by
combining ASR and MT into a single model. However, the usefulness of these models has only been investigated on language pairsof similar syntax and word order (e.g., English-French or EnglishSpanish). For syntactically distant language pairs (e.g., EnglishJapanese), speech translation requires distant word reordering.
Furthermore, parallel texts with corresponding speech utterancesthat are suitable for training end-to-end speech translation are
generally unavailable. Collecting such corpora is usually timeconsuming and expensive. This paper proposes the first attempt
to build an end-to-end direct speech-to-text translation system on
syntactically distant language pairs that suffer from long-distance
reordering. We train the model on English (subject-verb-object(SVO) word order) and Japanese (SOV word order) language
pairs. To guide the attention-based encoder-decoder model on thisdifficult problem, we construct end-to-end speech translation withtranscoding and utilize curriculum learning (CL) strategies that
gradually train the network for end-to-end speech translation
tasks by adapting the decoder or encoder parts. We use TTS fordata augmentation to generate corresponding speech utterancesfrom the existing parallel text data. Our experiment results showthat the proposed approach provides significant improvements
compared with conventional cascade models and the direct speechtranslation approach that uses a single model without transcoding
and CL strategies.
Index Terms—End-to-end speech-to-text translation, automatic
speech recognition, machine translation, multi-task learning.

S globalization continues to expand, language barriers
remain notorious obstacles to free communication. Spoken language translation is one innovative technology thatA

Manuscript received June 06, 2019; revised October 10, 2019, February 30,
2019 and March 6, 2020; accepted March 22, 2020. Date of publication April
5, 2020; This work was supported by the JSPS KAKENHI Grant Numbers
JP17H06101 and JP17K00237.
T. Kano is with the Division of Information Science, Nara Institute of Science and Technology, Ikoma 630-0192, Japan (e-mail:kano.takatomo.km04@is.naist.jp).
S. Sakti is with the Graduate School of Science and Technology, Nara
Institute of Science and Technology, Ikoma 630-0192, Japan, and also with
the RIKEN Center for Advanced Intelligence Project (ssakti@is.naist.jp).
S. Nakamura is with the Data Science Center and Graduate School of
Science and Technology, Nara Institute of Science and Technology, Ikoma
630-0192, Japan, and also with the RIKEN Center for Advanced Intelligence
Project (s-nakamura@is.naist.jp).

enables people to communicate among speakers of different
languages. However, translating spoken language remains avery complicated task that involves recognizing and automatically translating speech in real time.
A traditional approach in speech-to-speech translation systems constructs ASR, MT, and TTS systems that are trained
and tuned independently. Given speech input, ASR processes
and transforms the speech into text in the source language,
which MT then transforms into corresponding text in the target
language. Finally, TTS converts the target language text into
speech utterances [1]. The basic unit for information sharing
among these components is the “text representation” of words.
Even though significant progress has been made and various
commercial speech translation systems have been introduced,
this approach still suffers from several significant limitations.
One drawback is that over half of the world’s languages
are actually only spoken and have no written form. Thus,
constructing speech translation that heavily relies on information sharing of the text representation of words is difficult.
Another problem is that speech acoustics generally involve
both linguistic and paralinguistic information (i.e., rhythm,
emphasis, or emotion). Unfortunately, since such paralinguistic
information is not a factor in written communication, muchcannot be expressed in text. Consequently, the text output by
ASR has lost all paralinguistic information; only the linguistic
parts are translated by MT. Some studies have proposed theinclusion of additional components that just handle paralinguistic translation, but this step introduces more complexityand delay [2–4]. We need an architecture that can handle both
linguistic and acoustic feature contents at once and generatea translation to other languages.
Deep learning has shown much promise in many tasks. An
attention-based encoder-decoder neural network is a powerfulmodel for ASR, MT, and TTS [5–7]. Several recent works
have extended the task and constructed an end-to-end, directspeech-to-text translation system that combines ASR and MT
tasks in a single model. Duong et al. introduced the firststudy that considered speech-to-text translation with deepneural networks [8] and proposed alignment and translation
reranking directly from source-language speech with target
text translations. However, their work was only based on
Spanish-English language pairs with similar syntax and word
order, and the results failed to outperform the traditionalcascade approach based only on a statistical word level MT(MOSES) [9]. Their proposed attention-based model achieved
a BLEU score of 14.6%, where the MOSES baselines out-

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

performed their proposed method on the BLEU scores ina range between 18.2 and 20.2. Bérard et al. built a fullfledged, end-to-end attention-based speech-to-text translation
system [10]. It was the first work that used speech generated
by TTS as a corpus for speech translation (ST) tasks tosimplify the expansion of training data. However, this work
was only compared with statistical text-based MT systems andperformed on a small French-English synthetic corpus. These
two languages share similar word order (SVO-SVO). For suchlanguages, only local movements are sufficient for translation.
Bérard et al. [11] further investigated speech-to-text translation
with various units as character-, subword-, and word-based
speech translation with a beam search, a greedy search, andan ensemble search that combined them and successfully
improved the translation accuracy. Their studies concentrated
on French-English translation using both synthesis and natural
speech data. Weiss et al. focused on Spanish-English speechto-text translation and proposed sharing the parameters of an
ASR encoder to an ST encoder. Their study revealed that an
encoder could transform speech into a consistent interlingual
subword unit representation, which the respective decoders
assembled into phrases in either language [12]. Bansal et al.
then performed speech translation with natural speech frommulti-speakers and used unsupervised term discovery on cluster repeated patterns in the audio to create pseudo-text insteadof performing ASR [13]. Ultimately, all of these previous
researches just focused on syntactically similar language pairs.
In this work, we propose the first attempt to build an end-toend attention-based speech translation system on syntacticallydistant language pairs that suffer from long-distance reordering
phenomena. We trained our attention-based encoder-decoder
model on English-Japanese language pairs with SVO versus
SOV word order and utilized TTS for data augmentation to
generate the corresponding speech utterances from the existing parallel text data. To guide the attention-based encoderdecoder model to learn this difficult problem, we proposedtranscoding based on a curriculum learning (CL) strategy. Unlike a conventional CL strategy that starts with easy data andgradually emphasizes difficult data examples, we formalized
CL strategies that start the training with an end-to-end encoderdecoder for ASR or MT tasks and gradually trained the
network for end-to-end speech translation tasks by adapting
the decoder or encoder parts.
I. R ELATED W ORKS
Curriculum learning, which is a learning paradigm, wasinspired by the learning processes of humans and animals
that start by grasping easy aspects and gradually increasing tomore difficult ones. Although the application of such training
strategies to machine learning has been discussed between
machine learning and cognitive science researchers as far backas Elman et al. [14], CL’s first formulation in the context ofmachine learning was introduced by Bengio et al. (2009) [15].
Using CL might help avoid bad local minimums, hasten
training convergence, and improve generalization. These advantages have been empirically demonstrated in various tasks,
including shape recognition [15], object classification [16], and2language modeling [17]. However, most studies focus on how
to organize the sequence of the learning data examples in thecontext of single-task learning. Bengio et al. [15] proposed
CL for multiple tasks. Again, all of these tasks still belong
to the same type of problem (object classification) and shareidentical input and output spaces.
In speech translation tasks, the translation difficulty depends
on the relationship between source and target sentences, suchas word re-ordering, alignment, insertion, and deletion. Evenif both the source and target are clean speech and short
sentences, some cases are difficult to translate. For example,
some sentences might include uncommon names or words
with multiple definitions. These translation difficulties for ST
systems are found after training.
In contrast to most previous CL studies,(1) instead of utilizing the CL strategy for simple recognition/classification problems we use it for an attention-based
encoder-decoder neural network learning problems in speechtranslation tasks;(2) we train the model step by step from the easy task to
the complicated task changing the model structures. We start
training of an end-to-end encoder-decoder for ASR and MT
task. Then we gradually extend to the ST task by respectively
adapting the decoder or encoder parts; (3) in the original CL
learning, the input and output spaces are kept the same eventhe training data become difficult. However in our CL learning,the model input and output space will be changed as we go
to a more difficult task during the training steps.
II. BASIC ATTENTION - BASED S PEECH T RANSLATION
A. Attention-based encoder-decoder using RNN
We built an end-to-end speech translation system on a standard attention-based encoder-decoder neural network architecture using an RNN [6,18] that consists of encoder, decoder, andattention modules. Given input sequence x = [x1 , x2 , ..., xN ]with length N , the encoder produces a sequence of vector
enc
enc
representation henc = (henc
1 , h2 , ..., hN ). We used thefollowing bidirectional recurrent neural network (BiRNN)
with long short-term memory units (Bi-LSTM) [19] and agated recurrent unit (GRU) [20] that consists of forward andbackward recurrent neural networks (RNNs) [20] comprisedof forward and backward recurrent neural networks (RNNs):
hfn= LST M (xn )

hbn
henc
n

= LST M (xN −n )=(1)[hfn , hbn ].

Here hf denotes the forward RNN hidden states and hb
denotes the backward RNN hidden states. Thus, for each inputfxn , we obtain hencn by concatenating forward hn and backwardbhn . The decoder predicts target sequence y = [y1 , y2 , ..., yT ]with length T by estimating conditional probability p(y|x).
We use a uni-directional GRU. Conditional probability p(y|x)
is estimated based on the entire sequence of the previous
output:p(yt |y1 , y2 , ..., yt−1 , x) = softmax(Wy õdec
t ).

(2)

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

3

Decoder output vector odec
is computed by applying lineartlayer Wo to context information ct and current hidden state
hdec
t :
hdec
t

= GRU (embyt−1 )

ct
dec
ot

enc

= attention(h(3), hdect )= Wo [ct ; hdect ].

Here ct is the context information of the input sequencewhen generating the current output at time t, estimated by theattention module over encoder hidden states henc
n :
ct =

N
X

at (n) ∗ henc
n ,

(4)

n=1

where variable-length alignment vector at is computed whose
size equals the length of input sequence x:
at (n)=

decalign(hencn , ht )=dec
softmax(dot(hencn , ht )).

(5)This step helps the decoder find relevant information on theencoder side based on the current decoder hidden states. Theredec
are several variations for calculating align(hencn , ht ); we
simply use the general attention between the encoder anddecoder hidden states [21].
B. Attention-based encoder-decoder using Transformer
In performing RNN, since each step calculation needs
to weight the previous step process, the model cannot
compute the sequence data in parallel. Ashish et al. proposed
an attention-based encoder-decoder transaction model
without a recurrent mechanism called Transformer [22]. The
encoder maps an input sequence of symbol representations
x = [x1 , ..., xN ] to a sequence of continuous representations
h = [h1 , ..., hN ] using a stacked feed-forward neural network(FNN). Given h, the decoder generates output sequencey = [y1 , ..., yT ] of the symbols one element at a time.
Transformer follows this overall architecture using a stacked,
self-attention, point-wise FNN for both the encoder anddecoder. The encoder is composed of a stack of multiple
layers, each of which has two sub-layers. The first is amulti-head self-attention mechanism, and the second is aposition-wise FNN. Transformer has a residual connection
around each of the two sub-layers, followed by layer
normalization [23,24]. The decoder is also composed ofmultiple layers like the encoder. In addition to the two
sub-layers in each encoder layer, the decoder inserts a third
sub-layer, which performs multi-head attention over theencoder stack’s output. The attention function resembles
mapping a query and a set of key-value pairs to an output,
where the query, keys, values, and output are all vectors. The
output is computed as a weighted sum of the values, where theweight assigned to each value is computed by a compatibility
function of the query with the corresponding key. Fig.
1 illustrates the overall architecture of the Transformers.
Basically, the Transformer model offers two benefits: (1)it models sequential data without recurrent connections.
Therefore it does not need to wait to process previous results

Fig. 1. Overview of Transformer architecture [22].
and can process the whole sequence at once; (2) self-attention
provides an opportunity for injecting the global context
of the whole sequence into each input frame to directlybuild long-range dependencies. When performing forward
and backward processing for the current input and output,
Transformer only makes a calculation graph path for relative
states that are attended by a self-attention mechanism. But
the RNN-based encoder-decoder model makes a calculationgraph path for all the previous states because it models theglobal context with recurrent structures.

C. Basic attention-based speech translation
In this study, we apply this basic architecture to various
tasks.
• ASR system:
Input sequence x = [x1 , ..., xN ] is the input speech
sequence of the source language, and target sequencey = [y1 , ..., yT ] is the predicted text of the correspondingtranscription (Fig. 2). The RNN-based ASR system has
multi-layer perceptron (MLP) with a rectified linear
unit (ReLu) activation function as an input layer. The
encoder part consists of a Bi-LSTM at the 1st layer anda unidirectional GRU at the 2nd layer. Both networks
only use an even number of input vector sequences
to reduce the memory and calculation time. For the
decoder part, we use Luong’s design decoder [21] with a
GRU decoder. The Transformer-based ASR system has
an MLP and a convolution network as an input layer.
The encoder part consists of three fully connected (FC)layers and a self-attention function, and the decoder partIEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

consists of six FC layers and a self-attention function,
which are identical as the original Transformer encoderand decoder layers [22].

Fig. 2. ASR system•

Neural Machine Translation (NMT) system:
Input sequence x = [x1 , ..., xN ] is the text sequence ofthe source language, and target sequence y = [y1 , ..., yT ]is the predicted text sequence in the target language (Fig.
3). The text can be represented by words, subwords,
or character sequences. The RNN-based NMT encoder
part consists of two Bi-LSTM layers. For the decoder
part, we used Luong’s design decoder [21] with a GRU
decoder. The Transformer-based NMT encoder consistsof three FC layers and a self-attention function, and thedecoder part consists of six FC layers and a self-attentionfunction, which are identical to the original Transformer
encoder and decoder layers [22].

4

has identical architecture as each ASR model.

Fig. 4. Direct ST system– Cascade ST system:
The Cascade ST system combines two systems:
ASR and NMT (Fig. 5). The overall system has
input sequence x = [x1 , ..., xN ], which is the input
speech sequence of the source language and thetarget language sequence y = [y1 , ..., yT ]. We did
not perform joint training or adaptation for thismodel.

Fig. 3. NMT system

Fig. 5. Cascade ST system•

Speech translation (ST) system:– Direct ST system
Input sequence x = [x1 , ..., xN ] is the input
speech sequence of the source language, andtarget sequence y = [y1 , ..., yT ] is the predicted
corresponding text sequences (words, subwords, orcharacter sequences) in the target language (Fig.
4). As a baseline, we trained the attention-based
encoder-decoder neural network from an initialstate. Each RNN and Transformer-based direct ST– ASRenc-NMTdec ST system:
Input sequence x = [x1 , ..., xN ] is the input speech
sequence of the source language, and target sequencey = [y1 , ..., yT ] is the predicted corresponding text
(words, subwords, or character sequences) in thetarget language. Here the attention-based encoderdecoder neural network uses the pre-trained ASR
encoder and NMT decoder parts, following previous
works [11,12] (Fig. 6).

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

5

Fig. 6. ASRenc-NMTdec ST system

Fig. 8. End-to-end speech translation skipping odd index states

Transformer uses self-attention to model long-term context
information.
Fig. 7. Transcoding process– Proposed CL-Transcoder ST system:
Input sequence x = [x1 , ..., xN ] is the input speech
sequence of the source language, and target sequencey = [y1 , ..., yT ] is the predicted corresponding word
sequence in the target language. For the ST system
encoding and decoding parts, we pre-trained the
ASR encoder and NMT decoder networks. To
combine them smoothly to improve performance,
we used the transcoder to transfer the context vector
from the ASR encoder to the NMT decoder (Fig.
1). The transcoder architecture follows the NMT
encoder architecture. If we build an RNN-based
ST system, the transcoder should be a two-layer
Bi-LSTM. If we build a Transformer-based ST
system, the transcoder should be Transformer.
For the RNN-based ST and ASR encoder, since the input is
speech features (continuous space), we also did downsampling
during the encoding process to save memory resources [10].
The encoder’s first layer uses the entire input sequence. The
second layer only uses every other index, the third only uses
every fourth index, and so forth. The decoder maps input
embeddings to a hidden space using a unidirectional RNN
network and applies attention to get a context vector. Finally,
we concatenate the context and decoder hidden states and map
this super vector to the target language dictionary space usinga FC network [21]. Our end-to-end ST models also skip odd
index states, as shown in Fig. 8. However, we do not apply
this downsampling to the Transformer-based model, because

III. ATTENTION - BASED S PEECH T RANSLATION BYM ULTI - TASK L EARNING WITH T RANSCODING
Utilizing attention-based encoder-decoder architecture forconstructing a Direct ST task is difficult because the modelneeds to solve two complex problems:(1) learning how to process a long speech sequence andmapping it to the corresponding words, similar to the issues
focused on in the ASR field [5];(2) learning how to make proper alignment rules between thesource and target languages, similar to the issues discussed in
the NMT field [6,25]. In our proposed method, the attentionbased encoder-decoder neural network is not trained directlyfor speech translation tasks using parallel data. Inspired by
the original CL ideas, we proposed a new CL training strategy.
Instead of increasingly adding difficult data examples, as in theconventional CL, we train the model step by step from the easytask to the complicated task changing the model structures. We
start training of an attention-based encoder-decoder for speech
recognition (speech-to-text on the same source language) andtext-based machine translation (text-to-text on the source andtarget languages) task. Then we gradually extend to the endto-end speech translation (speech-to-text or speech-to-speechon the source and target languages) task. We describe eachtraining phase’s input and target sequence with their structuresin Figs. 9-11.
The encoder part received the speech features and converted
them by an MLP layer and output encoder state sequenceenc
H enc = [henc0 , ..., hn ]. We did the ASR decoding process to
get ASR decoder outputs hdec
m and used the MLP attention
mechanism for the RNN-based model and the Multi-headattention mechanism for the Transformer-based model. The

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

Fig. 9. Proposed: pre-training phaseFig. 10. Proposed: training transcoding phaseFig. 11. Proposed: total optimization phase6

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

ASR model provides a context vector sequence C ASR =
ASR
[cASR
0 , ..., cm ], where m represents the length of a sourcelanguage text (words, subwords, or character sequences):
hdec
m

=

am

=

cm

=

DecoderASR (embym−1 )
enc

(6)

, hdecm )attention(hNXat (n) ∗ hencn .

A. Experiments on BTEC data

The DecoderASR is a pre-trained ASR decoder, wherey denotes the source language-text (words, subwords, orcharacter sequences) and embym−1 denotes the previous ASR
decoder embedding vector. am is an attention score vector at
decoding step m. C ASR denotes a hidden representation of
the source-language acoustic information. We input source
text embedding X emb into the pre-trained NMT encoder andgenerated output as linguistic hidden states. The transcoding
process maps these acoustic hidden states to the linguistic
hidden states of the source language. This process improves
the attention result between the source and target sequencesin the NMT decoder. The transcoding receives ASR context
vector sequence C ASR and generates transcoder output H TRC .
The transcoder has identical architecture as the pre-trained
NMT encoder. We copied the pre-trained NMT encoder(except the embedding layer) and initialized the parameters.
We then used NMT encoder hidden states H NMT as a target
to optimize the transcoder:
H TRC = Transcoder(C ASR ),H= NMTEncoder(Xemblarge datasets to evaluate our proposed model and the baselinemodel. Finally, we used TED natural speech and performed
end-to-end speech translation to confirm the effectiveness of
our proposed approach.
IV. E XPERIMENTAL S ET-U P AND R ESULTS

n=1

NMT

7

(7)

).

Here X emb is a source-sentence embedding. The number ofH
and H NMT states equals the source text length. During
this transcoding process training, we froze the NMT encoder
parameters and only updated the ASR encoder and transcoder
parameters. We thoroughly optimized the transcoder to minimize the smooth L1 loss between H TRC and H NMT :
NMT 2
0.5 ∗ (hTRCm − hm ) ,



NMT
if |hTRC
m − hm | < 1,
(8)loss(H TRC , H NMT ) =
TRC
NMT

|hm − hm | − 0.5,otherwise.
TRC

The model can learn a difficult problem on a small dataset
using CL. End-to-end speech translation is difficult. Solving
this problem requires the preparation of a deeper neuralnetwork and larger amounts of data compared to regular
text NMT tasks. Since preparing parallel speech data is very
expensive, we start training a model on a simple task andproceed to a more difficult task. In this way, the difficulty ofthe problems gradually increases with each training phase. In
the end, the model can perform end-to-end speech translation
using only a small initial training dataset. We first performed
end-to-end speech translation with a small linguistic distantlanguage-pair dataset to confirm the CL benefits and compared
the translation performance of several model architectures. We
also performed speech translations with various language-pair

1) Experimental set-up for BTEC: First, we conducted
our experiments using a basic travel expression corpus(BTEC) [26,27]. The BTEC English-Japanese parallel corpus
consists of training (480k) and test (20k) utterances. The
BTEC English-French and Japanese-Korean corpus consistof training (160k) and test (500) utterances. We only usedutterances that exceed 4-words. Since the corresponding
speech utterances for this text corpus are unavailable, we used
the Google text-to-speech synthesis1 to generate a speech
corpus of the source language. To investigate the performanceof our proposed system in natural speech, we also utilized
the BTEC corpus that consists of 190k utterances of natural
English speech. Since it only has an 8k speech-to-text parallel
data of English-French and English-Japanese, we used naturaland generated speech to train the ASR and ST systems andtested it on BTEC natural speech data.
Throughout this experiment, we describe the benefits andpotential of our proposal’s results. First, we demonstrate the
BTEC translation task with the RNN-based model on thegenerated speech. Then we performed end-to-end translation
tasks on natural speech with the Transformer model [22,28],
which is a state-of-the-art sequential model. We changed all
the RNN networks to the FC layer and the self-attentionfunction and applied our proposed method to confirm how it
works with natural speech translation tasks. We segmented the
speech utterances into multiple frames with a 50-ms window
and 12-ms steps and extracted 80-dimension Mel-spectrogram
features using LibROSA2 . We further used these data to build
an attention-based ASR, an NMT system, the baseline Direct
ST system, and our proposed ST system. The hyperparametersettings of these models are displayed in Tables III-VIII.
TABLE IDATA SETTING OF BTEC GENERATED SPEECH GENERATED BY G OOGLETTSBTEC generated speech.
Language pairs
En-Ja
En-Fr
Paired speech
480k
160k

Ja-Ko
160k

TABLE IIDATA SETTING OF BTEC NATURAL SPEECHBTEC natural speech
Language pairs
En-Ja
Unpaired speech
190k
Paired speech
8k
1 Google TTS: https://pypi.python.org/pypi/gTTS
2 LibROSA: https://librosa.github.io/librosa/

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

For each system, we prepared characters, subwords [29],and words as translation sequences. At the evaluation steps,
our final goal is to increase the translation accuracy, whichis the word level. Therefore, we combined characters or
subwords into words for evaluation.
Next we summarize the network parameters. For all systems, we used the same learning rate and adopted Adam [30]in all of the models (Table VII).
We applied the attention-based encoder-decoder architecture
described in Section II to train the ASR, NMT, and Direct
ST systems. We also constructed a Cascade ST system and
an ASRenc-NMTdec ST system, as described in Figs. 56. For our proposed models, we applied our proposed CLbased training strategy to the attention-based encoder-decoderarchitecture described in Section III.
Baseline Cascade ST:
A conventional speech-to-text translation model thatcascades ASR and NMT systems (Fig. 5).
Baseline Direct ST:
A direct end-to-end speech translation model thatuses a single attention-based neural network (Fig. 4).
Baseline ASRenc-NMTdec ST:
An end-to-end speech translation model that uses a
pre-trained ASR encoder and an NMT decoder (Fig.
6).
Proposed CL-Transcoder ST:
Our proposed direct end-to-end speech translation
model trained with transcoder and CL strategies (Fig.
11).
To confirm our assumptions and the behavior of the proposed
method, we extracted a small dataset from the original dataset.
It was only 45k utterances for training and 500 utterances fortesting, and we also limited the length of the input speech
to less than 500 frames to save memory resources. Our ASR
system achieved an 8% word error rate (WER). For translation
quality, we compared the BLEU+1 scores of each model’s
performance. We chose BLEU+1 because a BTEC corpus
consists of many short utterances and BLEU+1 is a moresuitable objective evaluation method than BLEU [31] scoresfor short translations [32].
2) Experimental results on BTEC: First, we show how our
proposed method works during training with a small amountof data. In this experiment, we limited the training data to only
45 k of generated speech utterances. We report the validation
set softmax cross-entropy of each model in Fig. 12. From
this figure, we conclude that the direct speech translation
model encounters difficulties in the training process. This
leads us to suspect that we require more training data. On
the other hand, our proposed model and the pre-trained ASR
encoder and NMT decoder concatenation model reduced thevalidation loss even with fewer training data. Note also that
the ASRenc-NMTdec ST model’s first epoch validation loss
is as high as that of the direct translation model. Furthermore,
our transcoding method begins and converges with better
validation loss than the ASRenc-NMTdec ST model.
Table IX shows the translation results of the baseline andproposed systems with the BLEU+1 scores. We also include

8

TABLE IIIASR SETTINGSASR system
Input units
Downsampling ratio
MLP hidden units
Encoder RNN layers
LSTM and GRU hidden units
Encoder dropout ratio
Attention
Decoder layer depth
Decoder dropout ratio
Embed size
Embed dropout ratio

80
0.25
256
LSTM, GRU
256,5120, 0.3General
GRU
0.8
128
0.5

TABLE IVNMT SETTINGS

NMT system
Encoder layers
LSTM and GRU hidden units
Encoder dropout ratio
Attention
Decoder layer depth
Decoder dropout ratio
Embed size
Embed dropout ratio

LSTM, GRU
256, 512
01, 0.3
General
GRU
0.3
128
0.5

TABLE VD IRECT ST SETTINGS

ST system
Input units
Downsampling ratio
MLP hidden units
Encoder layers
LSTM and GRU hidden units
Encoder dropout ratio
Attention
Decoder layer
Decoder dropout ratio
Embed size
Embed dropout ratio

80
0.25
256
LSTM, GRU
256, 5120, 0.3General
GRU
0.3
128
0.5

TABLE VI
P ROPOSED ST SETTINGS

ST system
Input units
Downsampling ratio
MLP hidden units
Encoder layers
LSTM and GRU hidden units
Transcoder layers
LSTM and GRU hidden units
Decoder layer
Decoder dropout ratio
Embed size
Embed dropout ratio

80
0.25
256
LSTM, GRU
256, 512
LSTM, GRU
256, 512
GRU
0.3
128
0.5

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

9

Fig. 12. Small dataset of English-to-Japanese training steps with softmax cross-entropy loss

TABLE VIIO PTIMIZER SETTINGSTABLE IX
T RANSLATION RESULTS (BLEU+1) OF E NGLISH - TO -JAPANESELANGUAGE PAIRS WITH A SMALL DATASET

Optimization
Initial learning rate
0.001
Optimizing method
Adam [30]TABLE VIIIVOCABULARY SIZE OF EACH LANGUAGE AND SEGMENT ON BTEC

Language
EnglishJapanese
KoreanFrenchVocabulary size
Word
Subword
27011
2918
32794
2899
55092
2902
14802
2789

Character
29
2691
1422
31

text-to-text translation results (text-based NMT). The baseline
Direct ST system with a single attention module failed totranslate the English speech to Japanese text. Learning such
syntactically distant languages as Japanese and English isdifficult when the training data are limited. However, theperformance greatly improved when we applied pre-trained
ASR and NMT parameters to the encoder and decoder in the
ASRenc-NMTdec ST system, which achieved identical performance as the baseline Cascade ST systems. Our proposed
CL-Transcoder ST system achieved the best performance. It
can be stably trained and successfully outperformed all thebaseline systems with a significant BLEU+1 score margin.
The proposed method’s performance even surpassed that of
the text-based NMT. We constructed our proposed system with
[ASRenc+att]+[Transcoder]+[att+MTdec]. From a text-based
MT system viewpoint, the combination of the second and thirdparts resembles MT, and the additional components in the firstpart, which introduced more noise to the MT system’s input,
might function as a denoising encoder-decoder that prevents
overfitting.
Next we evaluated with a complete dataset and further
investigated the performance of the systems in various units(character, subword, and word units) and various language
pairs. We first calculated the WER for each ASR system

Model
Baseline Cascade ST
Baseline Direct ST
Baseline ASRenc-NMTdec ST
Proposed CL-Transcoder ST
Text-based NMT

BLEU+1
28.6
14.0
28.2
34.3
33.2

TABLE X
ASR WORD ERROR RATE ON SMALL AMOUNT OF BTEC DATA

Language
Ja
En

Characters
14.3
10.1

Subwords
7.1
6.0

Words
6.9
5.9

on a small BTEC dataset (Table X). Our ASR achieved asatisfactory performance below 10% WER. We achieved ahigher performance on ASR because we used speech generated
from TTS to train and evaluate the models. A single speakergenerated TTS speech, and the speaking style is very stable.
We then evaluated the translation quality for each systemand show the results in Tables XI-XIII. Tables XI and XII
demonstrate that the performances of the baseline Cascade ST
and Direct ST approaches are similar on subword and wordtranslation on syntactically similar language pairs. However,similar to the phenomena with the small dataset, the baseline
Direct ST did not perform well for syntactically distant language pairs. In such language pairs, richer architecture and abetter training strategy are necessary.
In contrast, our proposed models outperformed both baseline systems on syntactically distant language pairs in thecharacter-, subword-, and word-based systems (Table XIII).
Even on similar language pairs, our proposed approach successfully improved the end-to-end speech translation quality
in the subword and word units.
These experiments show that our proposed system has thepotential to outperform the Cascade ST model. However, theseresults are based on generated speech data; therefore, we also
performed a BTEC translation task with natural speech witha state-of-the-art sequential model Transformer for the ASR

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

10

0.8

0.6

0.4

0.2

0.0

Fig. 13. Japanese ASR attention

Fig. 14. Japanese-to-Korean direct translation attention

Fig. 15. Japanese-to-English cascade translation attention

Fig. 16. Proposed Japanese-to-English translation attention comparedwith cascade translation

Fig. 17. Japanese-to-English direct translation attention

Fig. 18. Proposed Japanese-to-English translation attention comparedwith direct translation

TABLE XI
BLEU+1 SCORE OF BASELINE C ASCADE ST SYSTEM

Language pairJa to EnJa to Ko
En to Fr
En to Ja

Characters
25.5
31.0
34.8
29.2

Subwords
30.5
40.1
39.9
32.7

Words
32.6
41.9
39.7
33.1

and NMT models. In this experiment, we used a character as abasic unit for ASR model. We prepared a BTEC English nat-

TABLE XIIBLEU+1 SCORE OF BASELINE D IRECT ST MODEL

Language pairJa to EnJa to Ko
En to Fr
En to Ja

Characters
17.3
29.6
22.3
20.4

Subwords
18.7
39.4
35.2
27.0

Words
19.3
39.0
36.8
22.3

ural speech dataset for English-French and English-Japanesetranslation. We summarize our Transformer parameters inIEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSINGTABLE XIIIBLEU+1 SCORE OF PROPOSED CL-T RANSCODER ST MODEL

Language pairJa to EnJa to Ko
En to Fr
En to Ja

Characters
30.5
30.1
33.0
30.9

Subwords
36.2
42.8
41.4
37.0

Words
37.4
43.0
42.7
38.6

Table XIX. We used the same Transformer model for ASR
and NMT, but ASR has a prenet module (FC and Convolution
network) instead of a source embedding layer [28]. We added
20% Gaussian noise to the decoder embedding vectors toincrease the beam-search performance on the test set. The
transcoder model has three FC layers and a self-attentionfunction, which is identical to the Transformer encoder layers.
First, we trained the ASR model using the BTEC natural andgenerated speech. ASR achieved a 6% WER on the BTEC
natural speech test set (500 utterances). The BTEC text NMT
systems were given English character sequences as input andoutput subword target sequences. We show our ASR model’s
performance in Table XIV and present the text NMT, CascadeST, and Proposed ST BLEU scores in Table XV. The results
for the RNN-based model in natural speech are slightly worse
compared to the performance with generated speech. But, if
we use the Transformer instead of RNN that is trained using
both natural and generated speech, we achieved a high ASR
performance of 6% WER. Also, the Transformer framework
improved the text translation performance when comparedto RNN-based model. Based on these results, we used theTransformer architecture as the standard architecture of our
proposed model in later sections.
TABLE XIVASR WORD ERROR RATE OF BTEC NATURAL SPEECHModel
RNN ASR
Transformer ASR

Test speech data
Generated
Natural
BTEC
BTEC
8%
9%
1%
6%

TABLE XVBLEU SCORES OF BTEC NATURAL SPEECH TRANSLATION

Model
Direct ST
ASR and NMT Cascade ST
End-to-end Proposed ST

En to Fr
36.2
40.3
43.8

En to Ja
28.2
35.0
40.0

B. Experiments on TED Talk data
1) Experimental set-up for TED Talk: We also performedexperiments on the TED corpus3 , which consisted of 270k
English sentences. All of these sentences have a corresponding
French translation, but only 210k have German translations.
3 TED talks: https://www.ted.com/talks

11

TABLE XVIDATA SETTING OF TED GENERATED SPEECH CREATED BY G OOGLE TTSSYSTEM

TED generated speech
Language pairs
En-Fr
En-De
Paired speech
270k
210k

TABLE XVIIDATA SETTING OF TED NATURAL SPEECHTED natural speech
Language pairs
En-Fr
En-De
Unpaired speech
236k
236k
Paired speech
8k
178k

TABLE XVIIIASR PRENET SETTINGS FOR TED TALKASR prenet functions
Input FC units
Output FC units
hline Convolution layers
Convolution input units
Convolution kernel size
Convolution dropout ratio
Batch normalize
Post FC units input and output units

80
512
3
512
5
0.2
True
512

TABLE XIXT RANSFORMER SETTINGS FOR TED TALK

Transformer parameters
Encoder layers
3
Decoder layers
6
Input and hidden units
512
Transformer FC units
1024
Multi-head number
8
Dropout ratio
0.2
TABLE XX
T EXT EMBEDDING SETTINGS FOR TED TALK

Text embedding layers
English character embedding input units
English subword embedding input units
German subword embedding input units
French subword embedding input units
Embedding output units
Position encoding
Decoder noising rate

32
31011
33124
31092
512
True
0.2

TABLE XXIO PTIMIZER SETTINGS

Optimization
Optimizing method
Adam [30]Warm-up steps
4000
Initial learning rate
0.001

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

We used these text datasets to train each English-French andEnglish-German MT model. For the speech-to-text model,
we used the TEDLIUM English 58k natural speech. Only
6k English utterances overlap between the TEDLIUM 58k
natural speech and the TED talk parallel text dataset. We madea 6k-utterance English-French and English-German natural
speech-to-text corpus using these overlapping data. To add
more data, we used Google TTS to generate another 270k
English speech utterances from a parallel text corpus. Finally,
we also utilized the IWSLT2018 English-German speech-totext TED dataset, which consists of 2k talk waveform data.
Based on the provided IWSLT2018 alignment information,
we segmented those waveforms and got 178k English-German
parallel utterances. Therefore, we trained our English ASR
and transcoder with the TEDLIUM 58k utterances of natural
speech, the IWSLT2018 178k utterances of natural speech,
and 270k generated speech utterances. The English-French
ST model was trained with a 270k generated speech-totext corpus and a 6k natural speech-to-text corpus, and theEnglish-German ST model was trained with a 210k generated
speech-to-text corpus, a 6k natural speech-to-text corpus, andthe IWSLT2018 178k utterances of the natural speech-to-text
corpus. For evaluation, we used the IWSLT2018 “dev2010”
dataset for a validation set as well as a “tst2015” and “tst2018”
datasets for test sets.
2) Experimental results on TED Talk: We also trained the
ASR model using TED natural speech, BTEC natural speech,
and TED generated speech. ASR achieved a 18% WER on
the TED natural speech test set shown in Table XXII. The
TED natural speech included lots of noise, and therefore
the generated speech and the BTEC natural speech did not
improve the TED natural test speech ASR performance. We
also trained the NMT model using only the TED corpus. We
present the text NMT, Cascade ST, and Proposed ST BLEU
scores in Table XXIII. For comparison in the same condition,
the TED text NMT systems were given English character
sequences as input and output subword target sequences. We
trained the text NMT for each language pair and chose thebest performance setting’s output segments to train the STmodel.

TABLE XXII
ASR WORD ERROR RATE OF TED NATURAL SPEECH

Model
Transformer ASR

tst2015
18 %tst2018
19 %TABLE XXIIIBLEU SCORES OF TED NATURAL SPEECH TRANSLATION

Model
Text-based NMT
Cascade ST
Proposed ST

En to Fr
tst2015
29.8
16.3
17.1

En to De
tst2015
tst2018
25.1
25.3
13.1
12.7
13.8
13.0

The results of this experiment are displayed in Table XXIII.
The performances of Cascade ST and Proposed ST models

12were affected by ASR errors. Although Transformer cangive better performance than that of RNN, the Transformer
still has some weaknesses. If the Transformer NMT got
incorrect inputs (i.e., ASR errors), then NMT may output
very short sequences (e.g., “so” or “and”). However, ourproposed method has a potential to recover the ASR errors
in the translation process. Therefore, our proposed method
outperformed Cascade ST in natural speech translation tasks.
C. Discussion
To provide more detail, we visualized a sample attention
matrix of alignment weights from the ASR and all translation
methods. Each pixel shows the alignment weight of the jth source sequence (in the horizontal axis) to the i-th target
sequence (in the vertical axis) in gray-scale (0: black indicatesthe lowest weight, and 1: white indicates the highest weight).
The attention weights of the ASR system are shown in Fig. 13.
The values are high along the diagonal of the matrix, which
illustrates the monotonous (left-to-right) natural alignment
between speech and text. Fig. 14 shows the attention weightsof the baseline Direct ST for similar language pairs of Japaneseand Korean. It also has a diagonal line, which indicatesthat the alignment of words between Japanese and Korean
is mostly monotonic. However, the attention scores are notas high as in the ASR system since there are many possible
ways to translate from one language to another. Therefore,
instead of having a significant weight in a single pixel, it has
weak weights in several pixels that represent several possible
candidates.
Next we investigated the attention weights for the ST taskwith distant language pairs. First, we compared the attention
weights of the baseline Cascade ST in Fig. 15 with theproposed CL-Transcoder ST in Fig. 16 for Sentence 1 ofTable XXIV. In our proposed method, we have two attention
modules. The first aligns the encoder’s hidden states of speechfeatures with the transcoder input sequence, and the secondaligns the transcoder output sequence with the decoder. In
this figure, we display only the second. The first attention
module has a monotonic shape that resembles ASR attention(Fig. 13). In the baseline Cascade ST, the NMT model input
is the sampled output from the ASR model hypotheses, which
is a single character, subword, or word unit. Unfortunately,when the ASR hypotheses are wrong, the NMT is unable
to fix such errors. On the other hand, our proposed CLTranscoder ST model solves this problem by directly mapping
the speech context vector into a latent translation state using
the transcoder. In this way, the speech context vector has
more latent information compared with the sampled characteror word. In other words, instead of having the 1-best ASR
hypothesis, it had an n-best hypothesis. As seen from Figs. 1516, our proposed method produces better attention and results,
specifically on the top part of attention weights “may I” versus“let me” due to the ability to recover some ASR errors. Thus,
our proposed CL-Transcoder ST with transcoder successfully
solved the error propagation problem in traditional model
cascade architecture.
We also compared the attention weights of the baseline
Direct ST in Fig. 17 with the attention weights of the proposed

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

13

TABLE XXIVT RANSLATION RESULTS FOR F IGS . 14-17

Recognition(Ja)
Translation(Ja to En)

Recognition(Ja)
Translation(Ja to En)

Speech translation results of sentence 1
Reference
seisan syo wo kakunin sa se te kuda sa i .
ASR Result
seisan syo wo kakunin shi ma su ka.
Reference
let me check your account .
Cascade ST
may i check my bill?
Proposed ST
let me check the bill.
Translation results of sentence 2
Referenceano otoko ha do ahoh de su .
ASR Result
ano otoko ha ** ahoh de su .
Referencethat man is an asshole.
Direct ST
the guy is in the tower of the tower of the blue man .
Proposed ST
the man is a horror man.

CL-Transcoder ST from the second attention module in Fig. 18
for Sentence 2 of Table XXIV. As seen from Fig. 17, there isno clear association between the speech input frames and theircorresponding words. This indicates that direct translation for
distant language pairs using standard attention-based end-toend architecture is difficult. In contrast, our proposed architecture addresses the problem with two attention modules and one
transcoder. The first attention handles the mapping between
the source speech encoder and the hidden representation of
the transcoder input, and the transcoder maps the underlying
representation of the input speech context vector into the latent
representation of the text transcription of the transcoder output.
Therefore, the task of the second attention is only to map
between the transcoder output state to the target text translationof the decoder. In this way, the attention module generated abetter attention map and result (Fig. 18) for Sentence 2 inTable XXIV.
Our most significant contribution is that we directly transferred the ASR hidden vector to the translation part. Since
the cascade models transfer the transcript results, when an
ASR error occurs, and attention spreads widely, the attention
module cannot find useful information to recover the ASR
error. In our proposed models, we processed the BiRNNs or
Transformer for the ASR context vectors in the transcoding
part, and the transcoder hidden states are the attention keys,
such as in the hidden states of machine translation encoders.
Therefore, when an ASR error occurs and the attention scores
spread widely, the attention module can gather information
before and after the hidden sequences to recover the error.
Another contribution is simplifying the task of the attention
module. Constructing a direct speech translation in a single
model means that we combine both tasks and perform manyto-many mapping tasks over long sequences (i.e., mappingfrom various speech utterances of identical contents to multiple
possible translations). For language pairs with similar syntax
and word order, the alignment between the source and target
sequences is almost monotonous (left-to-right), but the model
can still handle the task. In contrast, with syntactically distantlanguage pairs, speech translation requires distant word order.
Therefore the attention module suffers when it handles a long
input sequence and a distant alignment between the source andtarget sequence states. Such a condition is computationallyexpensive and often produces misalignment. Therefore, we
propose a framework with two attention modules and one
transcoder instead of just a single attention module. The
first attention module task works on the alignment between
a vector representation sequence of the speech utterance tothe corresponding text sequence in the source language andproduces context information (similar in complexity to an ASR
attention module). The second attention module task works
on the alignment between a vector representation of the text
sequence in the source language to the target language (similar
in complexity to an NMT attention module). Finally, thetranscoder task connects both attention modules and converts
the ASR context information to text vector representationin the source language. In this way, we can perform directspeech-to-text translation while keeping the complexity almost
similar to the cascade ASR-NMT systems.
Our final contribution is providing the possibility to perform
direct speech translation without large-size parallel speechdata. One main issue in developing a direct ST system isthe need for parallel speech (speech-text or speech-speechtranslation) data. Many studies have addressed this problem byutilizing TTS to generate speech data. In this paper, we alsouse 160k parallel text and the corresponding speech generated
by TTS for pre-training ASR and NMT as well as for the total
optimization of the direct ST system. This was done to havea fair comparison with the baseline Cascade ST and Direct
ST. Our proposed architecture, however, allows us to perform
without extensive size parallel speech data. For example, we
can pre-train ASR with natural and synthesized data that do
not have the corresponding translation and pre-train NMTwith standard parallel text that lacks relevant speech data. We
perform total optimization with only a small amount of parallel
natural speech data. Furthermore, our proposed method utilizes
pre-trained ASR and NMT models and transfers and optimizes
pre-trained models in a hidden state sequence as a transcoding
process. The optimization process no longer depends on the
text data of the source language.
Our second experiment on the BTEC and TED dataset
shows that our proposed method also works on difficult natural
speech translation tasks. The ASR and transcoder model canuse natural and generated speech corpus in training. The
transcoding process maps the ASR context sequence to theIEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

NMT encoder sequence; even then, however, ASR occurs,
and the context vector includes much noise. However, thetranscoding process can recover the noise. Therefore, ourproposed method outperforms the standard cascade ASR +NMT model.
If the source language lacks a written form or a sufficientamount of data, we can first pre-train the ASR and NMT
models using similar languages or other languages from thesame family that have written forms. We can then perform total
optimization with the required language. Further investigationis needed about the effectiveness of our proposed architecture
to extend the application to various languages (even for those
without a written form) as well as possibly extending the TTS
part and performing a complete speech-to-speech translation.
V. C ONCLUSIONSWe presented the construction of end-to-end speech translation for distant language pairs with transcoding based on CL
strategies that gradually trained the network for end-to-end
speech translation tasks by adapting decoder or encoder parts.
Our experimental results demonstrated that the translation
quality outperformed the Cascaded ST, standard Direct ST, and
ASRenc-NMTdec ST systems and revealed that our proposedmodel effectively decreased loss even using direct complex
problems. These results still rely on synthetic data because weneed to prepare the same data for all the baseline and proposedsystems, and a huge parallel speech corpus is not available.
Our proposed architecture, however, can effectively use monolingual speech and parallel text. Our transcoding process may
also be useful for joint TTS schemes to achieve speech-tospeech and/or paralinguistic translations. In the future, wewill investigate the effectiveness of our proposed method
using natural speech data and paralinguistic information andexpand the speech-to-text translation task to a speech-tospeech translation task.
R EFERENCES[1] S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro,
J. Zhang, H. Yamamoto, E. Sumita, and S. Yamamoto, “The ATR
multilingual speech-to-speech translation system,” IEEE Trans. Audio,
Speech & Language Processing, vol. 14, no. 2, pp. 365–376, 2006.
[Online]. Available: https://doi.org/10.1109/TSA.2005.860774[2] T. Kano, S. Takamichi, S. Sakti, G. Neubig, T. Toda,
and S. Nakamura, “Generalizing continuous-space translation ofparalinguistic information,” in INTERSPEECH 2013, 14th Annual
Conference of the International Speech Communication Association,
Lyon, France, August 25-29, 2013, F. Bimbot, C. Cerisara,
C. Fougeron, G. Gravier, L. Lamel, F. Pellegrino, and P. Perrier, Eds.
ISCA, 2013, pp. 2614–2618. [Online]. Available: http://www.iscaspeech.org/archive/interspeech_2013/i13_2614.html[3] Q. T. Do, S. Sakti, G. Neubig, and S. Nakamura, “Transferring emphasisin speech translation using hard-attentional neural network models,” inInterspeech 2016, 17th Annual Conference of the International SpeechCommunication Association, San Francisco, CA, USA, September
8-12, 2016, N. Morgan, Ed. ISCA, 2016, pp. 2533–2537. [Online].
Available: https://doi.org/10.21437/Interspeech.2016-898[4] P. D. Agüero, J. Adell, and A. Bonafonte, “Prosody generation forspeech-to-speech translation,” in 2006 IEEE International Conferenceon Acoustics Speech and Signal Processing, ICASSP 2006, Toulouse,
France, May 14-19, 2006. IEEE, 2006, pp. 557–560. [Online].
Available: https://doi.org/10.1109/ICASSP.2006.1660081

14[5] J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio,“Attention-based models for speech recognition,” in Advances in
Neural Information Processing Systems 28: Annual Conference on
Neural Information Processing Systems 2015, December 7-12, 2015,
Montreal, Quebec, Canada, C. Cortes, N. D. Lawrence, D. D. Lee,
M. Sugiyama, and R. Garnett, Eds., 2015, pp. 577–585. [Online].
Available: http://papers.nips.cc/paper/5847-attention-based-models-forspeech-recognition[6] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by
jointly learning to align and translate,” in 3rd International Conference
on Learning Representations, ICLR 2015, San Diego, CA, USA, May
7-9, 2015, Conference Track Proceedings, Y. Bengio and Y. LeCun,
Eds., 2015. [Online]. Available: http://arxiv.org/abs/1409.0473[7] Y. Wang, R. J. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss,
N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. V. Le,
Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron:
Towards end-to-end speech synthesis,” in Interspeech 2017, 18th
Annual Conference of the International Speech Communication
Association, Stockholm, Sweden, August 20-24, 2017, F. Lacerda, Ed.
ISCA, 2017, pp. 4006–4010. [Online]. Available: http://www.iscaspeech.org/archive/Interspeech_2017/abstracts/1452.html[8] L. Duong, A. Anastasopoulos, D. Chiang, S. Bird, and T. Cohn,“An attentional model for speech translation without transcription,”in NAACL HLT 2016, The 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human
Language Technologies, San Diego California, USA, June 12-17, 2016,
K. Knight, A. Nenkova, and O. Rambow, Eds. The Association for
Computational Linguistics, 2016, pp. 949–959. [Online]. Available:
https://doi.org/10.18653/v1/n16-1109[9] P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico,
N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer,
O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source
toolkit for statistical machine translation,” in ACL 2007, Proceedingsof the 45th Annual Meeting of the Association for Computational
Linguistics, June 23-30, 2007, Prague, Czech Republic, J. A. Carroll,A. van den Bosch, and A. Zaenen, Eds. The Association for
Computational Linguistics, 2007, pp. 177–180. [Online]. Available:
https://www.aclweb.org/anthology/P07-2045/[10] A. Berard, O. Pietquin, C. Servan, and L. Besacier, “Listen
and translate: A proof of concept for end-to-end speech-to-texttranslation,” CoRR, vol. abs/1612.01744, 2016. [Online]. Available:
http://arxiv.org/abs/1612.01744[11] A. Berard, L. Besacier, A. C. Kocabiyikoglu, and O. Pietquin,“End-to-end automatic speech translation of audiobooks,” in2018 IEEE International Conference on Acoustics, Speech andSignal Processing, ICASSP 2018, Calgary, AB, Canada, April
15-20, 2018. IEEE, 2018, pp. 6224–6228. [Online]. Available:
https://doi.org/10.1109/ICASSP.2018.8461690[12] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z. Chen, “Sequence-tosequence models can directly translate foreign speech,” in Interspeech
2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, F. Lacerda,
Ed. ISCA, 2017, pp. 2625–2629. [Online]. Available: http://www.iscaspeech.org/archive/Interspeech_2017/abstracts/0503.html[13] S. Bansal, H. Kamper, A. Lopez, and S. Goldwater, “Towards
speech-to-text translation without speech recognition,” in Proceedingsof the 15th Conference of the European Chapter of the Association
for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7,
2017, Volume 2: Short Papers, M. Lapata, P. Blunsom, and A. Koller,
Eds. Association for Computational Linguistics, 2017, pp. 474–479.
[Online]. Available: https://doi.org/10.18653/v1/e17-2076[14] S. Nolfi, D. Parisi, and J. L. Elman, “Learning and evolution in neuralnetworks,” Adaptive Behaviour, vol. 3, no. 1, pp. 5–28, 1994. [Online].
Available: https://doi.org/10.1177/105971239400300102[15] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum
learning,” in Proceedings of the 26th Annual International Conference
on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 1418, 2009, ser. ACM International Conference Proceeding Series, A. P.
Danyluk, L. Bottou, and M. L. Littman, Eds., vol. 382. ACM, 2009, pp.
41–48. [Online]. Available: https://doi.org/10.1145/1553374.1553380[16] C. Gong, D. Tao, S. J. Maybank, W. Liu, G. Kang, and J. Yang, “Multimodal curriculum learning for semi-supervised image classification,”
IEEE Trans. Image Processing, vol. 25, no. 7, pp. 3249–3260, 2016.
[Online]. Available: https://doi.org/10.1109/TIP.2016.2563981[17] A. M. Rush, S. Chopra, and J. Weston, “A neural attention
model for abstractive sentence summarization,” in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Language

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015,
L. Màrquez, C. Callison-Burch, J. Su, D. Pighin, and Y. Marton, Eds.
The Association for Computational Linguistics, 2015, pp. 379–389.
[Online]. Available: https://doi.org/10.18653/v1/d15-1044[18] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence tosequence learning with neural networks,” in Advances in Neural
Information Processing Systems 27: Annual Conference on Neural
Information Processing Systems 2014, December 8-13 2014, Montreal,
Quebec, Canada, Z. Ghahramani, M. Welling, C. Cortes, N. D.
Lawrence, and K. Q. Weinberger, Eds., 2014, pp. 3104–
3112. [Online]. Available: http://papers.nips.cc/paper/5346-sequenceto-sequence-learning-with-neural-networks[19] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” NeuralComputation, vol. 9, no. 8, pp. 1735–1780, 1997. [Online]. Available:
https://doi.org/10.1162/neco.1997.9.8.1735[20] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluationof gated recurrent neural networks on sequence modeling,” in NIPS
2014 Workshop on Deep Learning, December 2014, 2014. [Online].
Available: http://arxiv.org/abs/1412.3555[21] T. Luong, H. Pham, and C. D. Manning, “Effective approaches toattention-based neural machine translation,” in Proceedings of the 2015
Conference on Empirical Methods in Natural Language Processing,
EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, L. Màrquez,
C. Callison-Burch, J. Su, D. Pighin, and Y. Marton, Eds. The
Association for Computational Linguistics, 2015, pp. 1412–1421.
[Online]. Available: https://doi.org/10.18653/v1/d15-1166[22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N.
Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,”
in Advances in Neural Information Processing Systems 30: Annual
Conference on Neural Information Processing Systems 2017, 4-9
December 2017, Long Beach, CA, USA, I. Guyon, U. von Luxburg,
S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan,and R. Garnett, Eds., 2017, pp. 5998–6008. [Online]. Available:
http://papers.nips.cc/paper/7181-attention-is-all-you-need[23] L. J. Ba, R. Kiros, and G. E. Hinton, “Layer normalization,” CoRR, vol. abs/1607.06450, 2016. [Online]. Available:
http://arxiv.org/abs/1607.06450[24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in 2016 IEEE Conference on Computer Vision
and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June
27-30, 2016. IEEE Computer Society, 2016, pp. 770–778. [Online].
Available: https://doi.org/10.1109/CVPR.2016.90[25] P. Koehn, F. J. Och, and D. Marcu, “Statistical phrase-based translation,”in Human Language Technology Conference of the North American
Chapter of the Association for Computational Linguistics, HLT-NAACL
2003, Edmonton, Canada, May 27 - June 1, 2003, M. A. Hearst and
M. Ostendorf, Eds. The Association for Computational Linguistics,
2003. [Online]. Available: https://www.aclweb.org/anthology/N03-1017/[26] G. Kikui, E. Sumita, T. Takezawa, and S. Yamamoto, “Creating
corpora for speech-to-speech translation,” in 8th European Conferenceon Speech Communication and Technology, EUROSPEECH 2003
- INTERSPEECH 2003, Geneva, Switzerland, September 14, 2003.
ISCA, 2003. [Online]. Available: http://www.iscaspeech.org/archive/eurospeech_2003/e03_0381.html[27] G. Kikui, S. Yamamoto, T. Takezawa, and E. Sumita, “Comparative
study on corpora for speech translation,” IEEE Trans. Audio, Speech &Language Processing, vol. 14, no. 5, pp. 1674–1682, 2006. [Online].
Available: https://doi.org/10.1109/TASL.2006.878262[28] L. Dong, S. Xu, and B. Xu, “Speech-transformer: A norecurrence sequence-to-sequence model for speech recognition,”in 2018 IEEE International Conference on Acoustics, Speech andSignal Processing, ICASSP 2018, Calgary, AB, Canada, April
15-20, 2018. IEEE, 2018, pp. 5884–5888. [Online]. Available:
https://doi.org/10.1109/ICASSP.2018.8462506[29] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translationof rare words with subword units,” in Proceedings of the 54th
Annual Meeting of the Association for Computational Linguistics, ACL
2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
The Association for Computer Linguistics, 2016. [Online]. Available:
https://doi.org/10.18653/v1/p16-1162[30] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”in 3rd International Conference on Learning Representations, ICLR
2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings, Y. Bengio and Y. LeCun, Eds., 2015. [Online]. Available:
http://arxiv.org/abs/1412.6980[31] K. Papineni, S. Roukos, T. Ward, and W. Zhu, “Bleu: a method forautomatic evaluation of machine translation,” in Proceedings of the15

40th Annual Meeting of the Association for Computational Linguistics,
July 6-12, 2002, Philadelphia, PA, USA. ACL, 2002, pp. 311–318.
[Online]. Available: https://www.aclweb.org/anthology/P02-1040/[32] P. Nakov, F. Guzmán, and S. Vogel, “Optimizing for sentencelevel BLEU+1 yields short translations,” in COLING 2012, 24th
International Conference on Computational Linguistics, Proceedingsof the Conference: Technical Papers, 8-15 December 2012,
Mumbai, India, M. Kay and C. Boitet, Eds. Indian Instituteof Technology Bombay, 2012, pp. 1979–1994. [Online]. Available:
https://www.aclweb.org/anthology/C12-1121/[33] L. Màrquez, C. Callison-Burch, J. Su, D. Pighin, and Y. Marton, Eds.,
Proceedings of the 2015 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2015, Lisbon, Portugal, September
17-21, 2015. The Association for Computational Linguistics, 2015.
[Online]. Available: https://www.aclweb.org/anthology/volumes/D15-1/

Takatomo Kano received his B.E. from Doshisha
University, Kyoto, Japan, in 2011, and his M.S. fromthe Graduate School of Information Science, NAIST,
Nara, Japan in 2013. He is currently in the doctoral
course at NAIST, Japan. He interested in speechand natural language processing, with an end-to-end
speech translation. He is a student member of ISCA,
and ASJ.

Sakriani Sakti received her B.E. degree in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In 2000, she received
DAAD-Siemens Program Asia 21st Century Award
to study in Communication Technology, University
of Ulm, Germany, and received her MSc degreein 2002. During her thesis work, she worked with
Speech Understanding Department, DaimlerChrysler
Research Center, Ulm, Germany. Between 20032009, she worked as a researcher at ATR SLC Labs,
Japan, and during 2006-2011, she worked as anexpert researcher at NICT SLC Groups, Japan. While working with ATRNICT, Japan, she continued her study (2005-2008) with Dialog Systems Group
University of Ulm, Germany, and received her PhD degree in 2008. She actively involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003-2007), A-STAR and U-STAR (2006-2011).

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING

Satoshi Nakamura is Professor of Graduate Schoolof Information Science, Nara Institute of Science andTechnology, Japan, Honorarprofessor of Karlsruhe
Institute of Technology, Germany, and ATR Fellow.
He received his B.S. from Kyoto Institute of Technology in 1981 and Ph.D. from Kyoto University
in 1992. He was Associate Professor of Graduate
School of Information Science at Nara Instituteof Science and Technology in 1994-2000. He was
Director of ATR Spoken Language Communication
Research Laboratories in 2000-2008 and Vice president of ATR in 2007-2008. He was Director General of Keihanna ResearchLaboratories and the Executive Director of Knowledge Creating Communication Research Center, National Institute of Information and Communications
Technology, Japan in 2009-2010. He is currently Director of Augmented
Human Communication laboratory and a full professor of Graduate Schoolof Information Science at Nara Institute of Science and Technology. He isinterested in modeling and systems of speech-to-speech translation and speech
recognition. He is one of the leaders of speech-to-speech translation researchand has been serving for various speech-to-speech translation research projects
in the world including C-STAR, IWSLT and A-STAR. He received Yamashita
Research Award, Kiyasu Award from the Information Processing Society ofJapan, Telecom System Award, AAMT Nagao Award, Docomo Mobile Science Award in 2007, ASJ Award for Distinguished Achievements in Acoustics.
He received the Commendation for Science and Technology by the Minister of
Education, Science and Technology, and the Commendation for Science andTechnology by the Minister of Internal Affair and Communications. He also
received LREC Antonio Zampoli Award 2012. He has been Elected Board
Member of International Speech Communication Association, ISCA, sinceJune 2011, IEEE Signal Processing Magazine Editorial Board Member sinceApril 2012, IEEE SPS Speech and Language Technical Committee Member
since 2013, and IEEE Fellow since 2016.

16