SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM
WITH NEURAL INCREMENTAL ASR, MT, AND TTS
Katsuhito Sudoh, Takatomo Kano, Sashi Novitasari, Tomoya Yanagita, Sakriani Sakti, Satoshi Nakamura
Nara Institute of Science and Technology
8916-5 Takayamacho, Ikoma, 630-0192 Japan
ABSTRACTThis paper presents a newly developed, simultaneous neuralspeech-to-speech translation system and its evaluation. The
system consists of three fully-incremental neural processing
modules for automatic speech recognition (ASR), machinetranslation (MT), and text-to-speech synthesis (TTS). We investigated its overall latency in the system’s Ear-Voice Spanand speaking latency along with module-level performance.
Index Terms— simultaneous translation, neural speechto-speech translation, incremental speech recognition, incremental machine translation, incremental speech synthesis
1. INTRODUCTION
Computer-assisted cross-lingual conversation by automatic
speech-to-speech translation has been one of the most challenging problems in spoken language technologies in decades[1, 2]. Recent remarkable advances in speech and language
processing led by deep learning techniques benefit this challenge by real-time and accurate speech translation. One
crucial problem in automatic speech-to-speech translation
is its delay. Spoken language processing tasks are usuallyhandled in the utterance or sentence level. Their application
to the speech-to-speech translation suffers from a long delaythat is proportional to the input length, because the process
starts after the observation of the end of an utterance. Thatis similar to consecutive interpretation and is not useful for
long monologues such as lecture talks. On the other hand, insuch situations, simultaneous interpretation is often used foran audience not proficient in the language of a talk. Simultaneous interpretation is a challenging task to listen to the talkand speak its interpretation in a different language.
In this work, we tackle the problem of automatic simultaneous speech-to-speech translation and develop a neural system to do that from English to Japanese. Here, we call ourtask simultaneous translation, not simultaneous interpretation. We think the task of simultaneous interpretation includes
some additional efforts for summarization to make the output
concise for small latency and better understanding for the audience. The problem requires real-time and incremental processing for the output generated simultaneously with the in-put. Previous attempts to incremental neural speech translation focused on speech-to-text translation [3]. Our work aims
to speech-to-speech translation for natural information delivery by speech without a need for visual attention on text-based
subtitles. Our system is based on the cascade of three processing modules: incremental speech recognition (ISR), incremental machine translation (IMT), and text-to-speech synthesis (ITTS), rather than recent end-to-end approaches dueto the difficulty of applying them to the simultaneous translation.
We follow existing studies on incremental neural speechprocessing. For ASR, we choose an approach using a teacherstudent training framework to train an incremental student
model with the help of a non-incremental teacher model [4].
For MT, we choose an approach called wait-k, which delays
the start of the decoding process simply by k steps (i.e., k input
symbols from the ASR module) [5]. For TTS, we choose approach starting the segmental speech synthesis after observing the next accent phrase [6]. These modules exchange their
input/output symbols in the forms of subwords and work ina symbol-synchronous way, so they can be cascaded even if
they have different waiting strategies.
We also conduct a system-level evaluation of our system
in system-level latency and module-level performance on
English-to-Japanese simultaneous translation on TED Talks.
The system-level latency measures are: (1) processing delays
for waiting and computation time, and (2) TTS speaking latency derived from overlaps of synthesized speech outputs.
The module-level performance is measured by standard metrics in ASR, MT, and TTS. This work is the first attempt ofsystem-level evaluation for a simultaneous speech-to-speechtranslation system and would be beneficial to future studies.
2. SIMULTANEOUS SPEECH-TO-SPEECH
TRANSLATION
The largest motivation of simultaneous speech translation is
to convey spoken information in a different language withsmall latency. For example, in a lecture talk, a speaker oftentalks successively without any apparent pauses and sentenceboundaries. In such cases, a pause-based or sentence-based
processing suffers from large latency until a pause or sentence

boundary is observed. The latency may become much larger
in the latter part of a long talk due to accumulated processing
delays. Incremental processing helps catch up with the original speech. Human simultaneous interpretation is demanded
for the same reason.
Difficulty in simultaneous speech-to-speech translation issimilar to that in simultaneous human interpretation. In somelanguage pairs with a distant word order such as English andJapanese, human simultaneous interpretation is very challenging due to a large cognitive load to maintain long phrases
in short-term memory in the interpreter’s brain [7].
We show an example from the literature [7]. Suppose weare going to translate an English sentence:(1) The relief workers (2) say (3) they don’t have(4) enough food, water, shelter, and medical supplied (5) to deal with (6) the gigantic wave ofrefugees (7) who are ransacking the countryside(8) in search of the basics (9) to stay alive.
Its usual translation into Japanese is:(1) kyuuen tan’tousha wa (9) ikiru tame no (8)shokuryo o motomete (7) mura o arashi mawatteiru (6) tairyo no nan-min tachi no (5) sewao suru tame no (4) juubun na shokuryo ya
mizu, shukuhaku shisetsu, iyakuhin ga (3) nai
to (2) itte imasu.
where the numbered phrases correspond each other. Here we
can identify the verb “say” is translated into “itte imasu” atthe end of the Japanese translation. The following English
phrases are translated in the reverse order. This kind of translation causes a long delay and should be avoided for loweringinterpreters’ cognitive load and audience satisfaction. Experienced interpreters try to translate such a different input byrephrasing (refer to the literature for details.)
While the cognitive load itself would not be a serious
problem in a computer-based system, we still have to wait for
further input when we do not receive enough inputs to generate a partial translation. This problem lies in the MT phasein the whole simultaneous speech-to-speech translation workflow. ASR and TTS are regarded as sequential processes thatcan be solved in a left-to-right manner. The use of broad context has some benefits in the results, as revealed in previous
studies on incremental ASR and TTS. Thus, we employ incremental methods for all the processing modules and implement our simultaneous speech-to-speech translation system
by a cascade of these modules.
Given an input sequence X = x1 , x2 , ..., x|X| , a generalsequential transduction such as ASR, MT, and TTS predicts the corresponding output sequence Y = y1 , y2 , ..., y|Y | .
In incremental processing, the predictions are made separately on subsequences. Suppose an output subsequence
Y1k = y1 , y2 , ..., yk has been predicted from partial observations of the input X1i = x1 , x2 , ..., xi . When we predict the

Incremental ASR

Non-incremental ASR
Step n = 1

y1

y2

y3




yT

Decoder

y 1,1 y 1,2

h

LSTM

Step n = 2</m>y1 , K

Decoder

d
T

h

LSTM

Emb.

y 2,1

1

</m>

y2 , K

d
K1

2

h dK

2

Emb.

y0




y1

y2

yT-1

y 1 , K −1y 0 y 1,1 y 1 , K −1

aT

cT

Encoder

Encoder
BiLSTMh eS

BiLSTMaK

Attention
Matrix11
Attention

1cK

y 2 , K −1
2

aK

1

hWe

cK

2

...
2

h2e W

BiLSTM

BiLSTM

BiLSTM

BiLSTM

x1

x2

x3

x1

xS

x2

xW

xW+1 xW+2

x2W

Fig. 1. Incremental Speech Recognition system. [4]lnext output subsequence Yk+1
= yk+1 , ..., kl after furtherjpartial observations Xi+1 = xi+1 , ..., xj , the prediction ismade based on the following formula:jl
Yk+1= argmax P (Ŷ | X1i , Xi+1
, Y1k )

(1)Ŷ

where Ŷ is a possible prediction of the subsequence. We canuse an input and output history in an arbitrary length by recurrent and self-attention neural networks and do not needfurther approximation with a strong independence assumpl
tion on a Markov process. Note that the partial output Yk+1
jdoes not always corresponds directly to the observation Xi+1 .
The partial output may cover just a portion of the partial obj
servations Xi+1
, include some delayed predictions on the pastobservations X1i , or contain aggressive predictions more than
the given information by anticipations on future inputs.
One common problem here is how to decide the lengthof the partial observations to make partial predictions. The
most pessimistic way is to wait for the end of sentences, but
we must shorten such a delay for simultaneous translation.
In this work, we use fixed-length criteria for simplicity, as
presented in the next section. There have been some attempts
to determine it adaptively according to the inputs in MT [8, 9,10], we reserve the problem as future work.
3. INCREMENTAL PROCESSING MODULES
3.1. Incremental Automatic Speech Recognition
An ASR system usually starts its recognition process after
observing the end of a speech segment by voice activity detection. When the speech segment is long, the ASR transcription will be delayed together with its downstream processes
of MT and TTS. That causes serious latency in conversation
aided by the speech-to-speech translation system. To avoid
this problem, we implement an ISR following Novitasari etal.’s work [4], as shown in Fig.1. The ISR is trained witha teacher-student framework; a non-incremental ASR is pretrained as the teacher model, and then the ISR is trained asthe student model referring to the teacher model. The student model learns to mimic the alignment between the input
speech segments and the corresponding transcriptions givenby the teacher model, through attention transfer.
The teacher ASR model is used to transcribe an input
speech segment X = [x1 , . . . , xI ] (I: the number of timeframes of the segment) into a token sequence of the source
language S = [s0 , . . . , sN ] (s0 represents the beginning of asegment such as 
) and also provides a sequence of attentions A = [a1 , . . . , aN ], as follows:Han , ŝn= Encoder(X),(2)= Decoder(H, sn−1 ).

Here, we assume that the encoder is a bidirectional RNN
model and that the decoder is a unidirectional, attentional
RNN model. H is a sequence of the encoder hidden states.
ŝn is a predicted token, where n is an index in the sequence. We use the attention sequence A to split the input
speech X into shorter sub-segments, based on the assumption that the attention an gives an alignment between ŝnand its corresponding speech frames in X. Note that ŝn
is the same as sn in the training phase. As a result, thespeech segment X can be represented as a sequence of M
sub-segments, X̄ = [x̄1 , . . . , x̄M ]. We put an end-of-block
symbol <m> at the end of each sub-segment in the transcriptions S̄ = [s0 , S̄1 , . . . , S̄M ].
The student ISR model can be used to transcribe thespeech sub-segments X̄ incrementally into the corresponding
sub-segments of token sequences S̄, although it has the samearchitecture as the teacher ASR model. The ISR method alsouses look-ahead decoding that predicts outputs looking atlater frames. The look-ahead decoding requires additional
delays but improves recognition performance.
This ISR method achieved a 7.52% in character error rate
(CER) on a well-known WSJ dataset (WSJ SI-284 training
set and eval92 test set) [11]. In comparison, another work thatwas based on unidirectional long short-term memory (LSTM)
and connectionist temporal classification (CTC) [12] showed
CER of 10.96%.
3.2. Incremental Machine Translation
Machine translation has suffered from the problem of reordering, due to the syntactic difference between the source and target languages. The reordering problem is not so serious withcurrent attention-based neural MT technologies [13, 14, 15]
but still very problematic when we consider incremental processing, because we cannot observe full sentences. Several
previous studies tackled this problem by adaptive controls on
reading source language inputs and writing target language
outputs [16, 17]. In this work, we choose a recently-proposed
approach called wait-k, which delays the decoding process ink input tokens [5] as shown in Fig. 2, for simplicity.
The wait-k IMT model is used to translate a token sequence of the source language S = [s1 , . . . , sJ ] into that of

Incremental NMT

Non-incremental NMT
Step n=1

Step n = 2

Step n = 3
t1 t2 t3 tI
ℎI𝑑

LSTM

𝑡2

𝑡1
Decoder

Decoder

LSTM

Emb.

ℎ0𝑑

𝑡3
ℎ1𝑑2

ℎ2𝑑

Emb.

t0

s0 s1 s2 sI-1


Attention

𝑎𝐼

t1

𝑡2




𝑎0

𝑐𝐼

Encoder

Encoder
BiLSTMℎ𝑆𝑒

BiLSTM

𝑎1

𝑐0

𝑎2

𝑐1

ℎk𝑒

𝑐2

𝑒
ℎ𝑘+1

...
𝑒
ℎ𝑘+2

BiLSTM

BiLSTM

BiLSTM

BiLSTMs1 s2 s 3

s1 s2

sJ

sk

s1

s2

sk+1

s1

s2

sk+2

Fig. 2. Incremental Machine Translation system. [5]
DNN-ITTS

Accent phrase Accent
sequencesphrase 1...

Accent phrase
detection...

Word sequences Word 1
POS 1
with POS tag

Word 2
POS 2 . . .

Incremental . . .
POS tagger
Word sequencesfrom IMT

Word 1

Word 2...

Fig. 3. Incremental text-to-speech synthesis system.
the target language T = [t1 , . . . , tI ] as follows.
Hi=

Encoder(s1 , . . . , si+k−1 ),t̂i=

Decoder(Hi , t̂1 , . . . , t̂i−1 ).

(3)

The decoder has to predict an output token based on the attention over an observed portion of the input tokens. k is a
hyperparameter for the fixed delay in this model; setting klarger causes longer delays, while smaller k would result inworse output predictions due to the poor context information.
This IMT method with k = 5 achieved 21.53 in BLEU
on a common ASPEC English-Japanese dataset [18]. Its nonincremental counterpart resulted in 32.22 in BLEU, mainly
due to the word order differences mentioned in section 2.
3.3. Incremental Text-to-Speech Synthesis
The workflow of our ITTS module is illustrated in Fig. 3.
Since Japanese is a mora-timed language, This module isbased on a mora-timed Japanese ITTS framework [19]. A
mora is a sub-unit syllable with one short vowel and any preceding onset consonant that corresponds to a single Japanesehiragana phonogram [20]. An accent phrase is a mora sequence longer than a word. The Japanese accent of each mora

4. EVALUATION
Vocoder
Acoustic
featureAcoustic
LSTM
Frame-level
Linguisticfeatures
Duration
LSTM
Phoneme-level
linguistic features

We conducted a system-level evaluation of our system by an
English-to-Japanese simultaneous speech-to-speech translation experiment. The focuses in the evaluation were: (1)System-level latency and (2) Module-level quality, to investigate the detailed performance of the system in practice.
4.1. Evaluation Setup

Linguistic feature extraction
Text analysis
Accent phrase

Fig. 4. DNN-based TTS system. [22]indicates the relative pitch change in the accent phrase andhas an important role in the prosody, similar to the tone typein tonal languages [21].
Yanagita et al. [19] synthesized speech outputs by accent phrases using linguistic features on the input text. Our
ITTS module is different from theirs as follows: (1) we synthesize an output for a single accent phrase at a time without grouping it with future accent phrases; (2) we exclude
part-of-speech-based features in the synthesis step for simplicity; (3) we choose a neural approach with LSTMs insteadof HMMs. Fig. 4 illustrates the ITTS built upon a DNN TTS
framework [22]. It uses an accent phrase as a basis for generating a waveform through incremental part-of-speech tagging
and rule-based re-segmentation on incremental inputs fromthe IMT (Fig 3). It predicts the waveform for an accent phrase
given by the preprocessing. The waveform prediction consistsof the following subprocesses, as illustrated in Fig 4.
1. Linguistic features are extracted from the input accentphrase, in forms of a sequence of phonemes and accent
information. Accent information consists of phoneme
and accent type, their position in the accent phrase, andthe number of moras in the accent phrase.
2. Phoneme-level one-hot vectors are generated using the
linguistic features.
3. Phoneme durations are predicted using duration LSTMs.
4. Frame-level vectors are generated from the one-hot
vectors based on the phoneme durations. Positional information is concatenated with the frame-level vectors.
5. Acoustic features are predicted using acoustic LSTMs
and the frame-level vectors.
6. A waveform is generated by the WORLD [23] (D4C
edition [24]) vocoder from the acoustic features.

We trained the ISR, IMT, and ITTS models with the settings
shown in Table 1 and the following training procedures.
For ISR training, we used TED-LIUM release 1 [25]. We
extracted 80 dimension mel-spectrogram feature as an input.
The English speech transcriptions used in the training of ISR
were segmented into subwords by SentencePiece [26]. The
English SentencePiece model was shared with the IMT part.
The IMT model was LSTM-based attentional encoderdecoder model, trained using 1.41 million English-Japaneseparallel sentences from WIT3 [27] The SentencePiece model
was trained using the training sentences and the default training options other than the vocabulary size. The subword vocabulary size was 8,000 for English and 32,000 for Japanese.
For ITTS model training, we utilized one female Japanese
speech dataset [28] with 8k pairs of speech utterances and thecorresponding transcriptions. We used 5k utterances for the
training set, 100 utterances for the development set, and 100
utterances for the test set. Here we extracted 39 dimensionsof Mel-generalized cepstral feature, 1-dimension of continuous log f0, 2-dimension of band averaged aperiodicity and 1dimension of voiced/unvoiced flag as a target. The frame period was set to 5ms. We used a modified Open Jtalk1 system
for the text analysis system. We trained an incremental partof-speech tagger2 using the Corpus of Spontaneous Japanese(CSJ) dataset [29]. Phoneme duration was extracted forcedalignment by HTS system3 and linguistic features.
The three modules were running in parallel and connected
via UNIX pipes; an output by a module was sent to stdout,
and it was passed into stdin of the next module. The system
could be implemented in a server-client framework, but weused this naı̈ve implementation in this paper.
The test set consisted of two TED talks (Talk1 13:32,
Talk2 17:19) with English and Japanese subtitles and were
not included in WIT3. We could conduct a practical evaluation using a single input speech stream, however, we used apause-based segmented speech because we observed significant speaking latency by TTS outputs as presented later in
4.2.2. Note that the modules were implemented to flush its
hidden vectors and be initialized when the end-of-sequence
symbol 
 was observed.
1 http://open-jtalk.sourceforge.net/
2 https://github.com/odashi/incremental-tagger
3 http://hts.sp.nitech.ac.jp/

Table 1. Model setting of the modules.
ISR settings
Input feature size
Output vocabulary size
Encoder input feed-forward sizes
Decoder embedding size
Encoder/Decoder RNN hidden size
# Encoder/Decoder RNN layers
Attention function
IMT settings
Input/Output vocabulary size
# Encoder/Decoder layers
RNN hidden size
Delay ”k”
ITTS settings
Duration / acoustic model input size
Duration / acoustic model output size
Hidden size
# RNN layers

80
8,009
512
256
256 / 512
3/1
Markov history MLP [30]8,000 / 32,000
2/2
512
5

Table 3. Evaluation results of Incremental Speech Recognition by word error rate (WER) and character error rate (CER).
Talk
WER
CER
Talk 1 0.344 0.199
Talk 2 0.221 0.117

From Table 2, we can see that the ITTS module brought
long delays, and the delays were highly influenced by different inputs. The delays and their variance in the IMT module
were less than those by the ITTS, and the ISR module worked
with almost fixed delays.
4.2.2. Speaking latency

297 / 300
1 / 44512
1

Table 2. Module-level processing delays from the beginningsof speech inputs by the ISR, IMT, and ITTS. Note again later
modules have to work after their preceding module, so theirdelays include the delay in the preceding module.
ModuleMean
Variance
ISR delay
1.24 (sec)
0.031
IMT delay
2.04 (sec)
0.192
ITTS delay 3.218 (sec)
0.377

4.2. System-level Latency
We evaluated the latency of our system using two metrics:
Ear-Voice Span (EVS) and Speaking latency.
4.2.1. Ear-Voice Span (EVS)
EVS is a common latency measure in the field of simultaneous interpretation. EVS measures the delay between thestart of a source speech and the start of the corresponding
interpretation speech. We measured the system-level delay
in simultaneous speech-to-speech translation in a similar way
for a human interpreter, using the delay by the module-levelprocessings and inter-module communications, which we callsystem-level EVS.
For the system-level EVS evaluation, we aligned the ISR,
IMT, and ITTS outputs with their output time information,
as shown in Fig. 5. The mel features were segmented in every 32 frames (0.55 seconds). The ISR delayed three blocks,
IMT delayed five blocks, and ITTS delayed seven blocks inthis example. Both of ISR and IMT delays are controlled by
the hyper-parameters. The ITTS delay depends on the output
timing of the accent phrase generator. We averaged modulelevel delays in the test set speech that was further segmented
into 170 short streams. The delays are shown in Table 2.

We found large delays derived from ITTS outputs in oursystem development. A speaker of a TED-like talk speaks
densely and does not put long pauses. Thus, a new input cameinto the ITTS module even when the previous output waveform was being played, as shown in Fig. 5; the latter parts of
TTS outputs were queued and played with large delays. Thisoverlap would be a very serious problem in a long stream ofthe speech input. So, we employed the segment-based evaluation in this work and considered the talk-level evaluation as
future work.
4.3. Module-level Quality
We evaluated the module-level quality for all the three modules: ISR, IMT, and ITTS. In the subjective evaluation described later, we asked eleven people (graduate school students) to evaluate the results on a 1-5 scale.
4.3.1. Incremental Speech Recognition
We evaluated ISR results with word error rate (WER) and
CER. Table 3 shows the results. There were large differences
between the two talks.
4.3.2. Incremental Machine Translation
We evaluated ISR+IMT results with BLEU-{1,2,3,4} [31] using the reference from the corresponding voluntary translated
Japanese subtitles. Table 4 shows the results by SacreBLEU4[32]. Unfortunately, the BLEU-4 score by the ISR+IMT
translation was very low. There would be various reasons, such as: (1) Domain mismatch; the amount of indomain training sentences was not large, (2) Style mismatch;Japanese translations in TED Talks subtitles are usually not
literal ones, so the surface-based evaluation by BLEU may
not work well, and (3) ASR error propagation.
We also conducted a subjective evaluation of the ISR+IMT
results. The subjective evaluation was based on 1-5 scale
Adequacy and Fluency metrics used in past MT evaluation
4 Fingerprint:
BLEU+case.mixed+lang.en-ja+numrefs.1+smooth.exp+
tok.ja-mecab-0.996-IPA+version.1.4.13

ISR & IMT &ITTS feature synthesis
DoneTime (sec)
ISR
IMT

0.0

0.8

1.2
1.6
2
2.4
how many companies have you
interact
Dore kurai no

ITTS

2.8
ed
Kaisha

3.2
with
ga

Dore-kurai-no

ISR delay

IMT delay

3.6
today
yaritori

ITTS speaking
Done

4


wo | shiteiru ka shimeshi te iru |

5.6

6

6.4

6.8

kaisha-ga-yaritori |wo shite iruka shimeshi teiru |ITTS delay

Fig. 5. ISR, IMT, and ITTS output and alignment example.

Table 4. Evaluation results of Incremental Machine Translation by BLEU-{1,2,3,4} and the length ratio between the
hypothesis and referernce.
Talk
BLEU
Length
1
2
3
4
ratio
Talk 1 26.2 12.9 6.6 3.5
1.161
Talk 2 29.6 14.8 8.2 4.9
1.127
Table 5. Subjective evaluation results of Incremental Machine Translation by Adequacy and Fluency in a 1-5 scale.
Average and standard deviation values are shown.
Talk
Adequacy
Fluency
Talk 1 1.76 (0.86) 2.54 (1.25)
Talk 2 1.94 (0.87) 2.48 (1.26)

campaigns [33]. Table 5 shows the results. As expected fromthe ISR and BLEU results, the adequacy result on Talk 2 wasslightly better than that on Talk 1.

Table 6. Subjective evaluation results of Incremental Speech
Synthesis by Mean Opinion Score (MOS) in a 1-5 scale. Average and standard deviation values are shown.
Talk
MOS
Talk 1 2.34 (0.87)
Talk 2 2.55 (1.01)

En subISR
Ja sub
IMT
En subISR
Ja subIMT

Table 7. Translation examples.
here’s another viewpoint of the world
here’s another point of the world
mou hitotsuno sekai no mikata oohanashi shimasu
kore wa betsu no shiten deno shiten desu
(This is a viewpoint from another viewpoint.)
think about this dog here
think about this dark here
kono inu o rei ni tori masu
kono daaku enerugii o kangaete mite kudasai(Please think about this dark energy)

4.3.3. Incremental Speech Synthesiss
Our ITTS evaluation was only with subjective evaluation, because we do not have a reference speech in this task. The subjective evaluation was based on Mean Opinion Score (MOS),
focusing on the naturalness of the output speech rather thancontents. Table 6 shows the results.
5. DISCUSSION
The latency results revealed that our incremental system
based on the cascade of three modules worked successfully
with relatively small delays. However, the quality results suggested the task difficulty due to error propagation from ISR to
IMT and lack of in-domain corpora in English-Japanese MT.
We show two translation examples in Table 7. The first one
is a relatively good result, and the other one is a typical error
propagation example. Tight module integration would bepromising, such as lattice-to-sequence [34], but its extension
to the simultaneous translation is not trivial.
Besides, we have no common evaluation metrics for simultaneous speech-to-speech translation other than modulelevel ones. We used two latency metrics in this work, but

the objective measurement of content delivery by speech-tospeech translation is crucial for further evaluation.
6. CONCLUSIONSIn this paper, we presented our English-to-Japanese simultaneous speech-to-speech translation system with its evaluation
using English TED talks. The system works fully-incremental
with speech inputs, by the cascaded modules of incremental ASR, incremental MT, and incremental TTS. The latency
evaluation revealed the module-level computation could befinished with about three seconds delay at maximum. However, the system suffers from speaking latency.
Our future work includes further improvement of themodules in accuracy and efficiency, and controlling speaking
duration to decrease the speaking latency.
7. ACKNOWLEDGMENTS
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101.

8. REFERENCES[1] Eiichiro Sumita, Tohru Shimizu, and Satoshi Nakamura,“NICT-ATR speech-to-speech translation system,” inProceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume
Proceedings of the Demo and Poster Sessions, Prague,
Czech Republic, June 2007, pp. 25–28, Association for
Computational Linguistics.
[2] Sakriani Sakti, Michael Paul, Andrew Finch, Xinhui Hu,
Jinfu Ni, Noriyuki Kimura, Shigeki Matsuda, ChioriHori, Yutaka Ashikari, Hisashi Kawai, Hideki Kashioka, Eiichiro Sumita, and Satoshi Nakamura, “Distributed speech translation technologies for multiparty
multilingual communication,” ACM Trans. SpeechLang. Process., vol. 9, no. 2, Aug. 2012.
[3] Jan Niehues, Ngoc-Quan Pham, Thanh-Le Ha, Matthias
Sperber, and Alex Waibel, “Low-latency neural speechtranslation,” in Proceedings of Interspeech 2018, 2018,
pp. 1293–1297.
[4] Sashi Novitasari, Andros Tjandra, Sakriani Sakti, andSatoshi Nakamura, “Sequence-to-Sequence Learning
via Attention Transfer for Incremental Speech Recognition,” in Proceedings of Interspeech 2019, 2019, pp.
3835–3839.
[5] Mingbo Ma, Liang Huang, Hao Xiong, Renjie
Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang,
Zhongjun He, Hairong Liu, Xing Li, Hua Wu, andHaifeng Wang, “STACL: Simultaneous translation
with implicit anticipation and controllable latency using
prefix-to-prefix framework,” in Proceedings of the 57th
Annual Meeting of the Association for Computational
Linguistics, Florence, Italy, July 2019, pp. 3025–3036,
Association for Computational Linguistics.
[6] Tomoya Yanagita, Sakriani Sakti, and Satoshi Nakamura, “Neural iTTS: Toward Synthesizing Speech
in Real-time with End-to-end Neural Text-to-Speech
Framework,” in Proceedings of the 10th ISCA Speech
Synthesis Workshop, 2019, pp. 183–188.
[7] Akira Mizuno, “Simultaneous Interpreting and Cognitive Constraints,” Journal of College of Literature,
Aoyama Gakuin University, vol. 58, pp. 1–28, 2016.
[8] Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang
Huang, “Simpler and faster learning of adaptive policies for simultaneous translation,” in Proceedings ofthe 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLPIJCNLP), Hong Kong, China, Nov. 2019, pp. 1349–
1354, Association for Computational Linguistics.

[9] Katsuki Chousa, Katsuhito Sudoh, and Satoshi Nakamura, “Simultaneous neural machine translation using
connectionist temporal classification,” arXiv preprint,
vol. abs/1911.11933, 2019.
[10] Baigong Zheng, Kaibo Liu, Renjie Zheng, Mingbo Ma,
Hairong Liu, and Liang Huang, “Simultaneous translation policies: From fixed to adaptive,” in Proceedings of
the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020, pp. 2847–2853,
Association for Computational Linguistics.
[11] Douglas B Paul and Janet M Baker, “The design for thewall street journal-based csr corpus,” in Proceedings of
the workshop on Speech and Natural Language. Association for Computational Linguistics, 1992, pp. 357–362.
[12] Kyuyeon Hwang and Wonyong Sung, “Character-level
incremental speech recognition with recurrent neuralnetworks,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2016, pp. 5335–5339.
[13] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “Neural Machine Translation by Jointly Learning
to Align and Translate,” in Proceedings of the International Conference on Learning Representations (ICLR
2015) arXiv:1409.0473, 2015.
[14] Thang Luong, Hieu Pham, and Christopher D. Manning,“Effective approaches to attention-based neural machinetranslation,” in Proceedings of the 2015 Conferenceon Empirical Methods in Natural Language Processing,
Lisbon, Portugal, Sept. 2015, pp. 1412–1421, Association for Computational Linguistics.
[15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser,
and Illia Polosukhin, “Attention is All you Need,” inAdvances in Neural Information Processing Systems 30,
pp. 5998–6008. Curran Associates, Inc., 2017.
[16] Tomoki Fujita, Graham Neubig, Sakriani Sakti, Tomoki
Toda, and Satoshi Nakamura, “Simple, Lexicalized
Choice of Translation Timing for Simultaneous Speech
Translation,” in Proceedings of Interspeech 2013, 2013,
pp. 3487–3491.
[17] Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li, “Learning to translate in real-time withneural machine translation,” in Proceedings of the 15th
Conference of the European Chapter of the Association
for Computational Linguistics: Volume 1, Long Papers,
Valencia, Spain, Apr. 2017, pp. 1053–1062, Association
for Computational Linguistics.

[18] Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara, “ASPEC: Asian scientificpaper excerpt corpus,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia, May 2016, pp.
2204–2208, European Language Resources Association(ELRA).
[19] Tomoya Yanagita, Sakriani Sakti, and Satoshi Nakamura, “Incremental tts for japanese language.,” in Proceedings of Interspeech 2018, 2018, pp. 902–906.
[20] Masayuki Suzuki, Ryo Kuroiwa, Keisuke Innami,
Shumpei Kobayashi, Shinya Shimizu, Nobuaki Minematsu, and Keikichi Hirose, “Accent sandhi estimationof tokyo dialect of japanese using conditional random
fields,” IEICE TRANSACTIONS on Information and
Systems, vol. 100, no. 4, pp. 655–661, 2017.
[21] Shuji Yokomizo, Takashi Nose, and Takao Kobayashi,“Evaluation of prosodic contextual factors for HMMbased speech synthesis,” in Eleventh Annual Conference of the International Speech Communication Association, 2010.
[22] Heiga Zen and Hasim Sak, “Unidirectional long shortterm memory recurrent neural network with recurrent
output layer for low-latency speech synthesis,” in 2015
IEEE International Conference on Acoustics, Speechand Signal Processing, ICASSP 2015, South Brisbane,
Queensland, Australia, April 19-24, 2015. 2015, pp.
4470–4474, IEEE.
[23] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa,“World: a vocoder-based high-quality speech synthesis
system for real-time applications,” IEICE TRANSACTIONS on Information and Systems, vol. 99, no. 7, pp.
1877–1884, 2016.
[24] Masanori Morise, “D4c, a band-aperiodicity estimatorfor high-quality speech synthesis,” Speech Communication, vol. 84, pp. 57–65, 2016.
[25] Anthony Rousseau, Paul Deléglise, and Yannick Estève,“TED-LIUM: an automatic speech recognition dedicated corpus,” in Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC. 2012, pp. 125–129, European Language
Resources Association (ELRA).
[26] Taku Kudo, “Subword regularization: Improving neuralnetwork translation models with multiple subword candidates,” in Proceedings of the 56th Annual Meeting of
the Association for Computational Linguistics (Volume
1: Long Papers), Melbourne, Australia, July 2018, pp.
66–75, Association for Computational Linguistics.

[27] Mauro Cettolo, Christian Girardi, and Marcello Federico, “WIT3 : Web Inventory of Transcribed and Translated Talks ,” in Proceedings of the 16th EAMT Conference, 2012, pp. 261–268.
[28] Ryosuke Sonobe, Shinnosuke Takamichi, and Hiroshi
Saruwatari, “JSUT corpus: free large-scale Japanese
speech corpus for end-to-end speech synthesis,” arXiv
preprint, vol. abs/1711.00354, 2017.
[29] Kikuo Maekawa, Hanae Koiso, Sadaoki Furui, and Hitoshi Isahara, “Spontaneous speech corpus of japanese,”
in Proceedings of the Second International Conferenceon Language Resources and Evaluation, LREC. 2000,
European Language Resources Association.
[30] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura,“Multi-scale alignment and contextual history for attention mechanism in sequence-to-sequence model,” in2018 IEEE Spoken Language Technology Workshop,
SLT. 2018, pp. 648–655, IEEE.
[31] Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu, “Bleu: a method for automatic evaluation ofmachine translation,” in Proceedings of the 40th Annual
Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, July 2002, pp.
311–318, Association for Computational Linguistics.
[32] Matt Post, “A call for clarity in reporting BLEU scores,”in Proceedings of the Third Conference on MachineTranslation: Research Papers, Brussels, Belgium, Oct.
2018, pp. 186–191, Association for Computational Linguistics.
[33] LDC, “ Linguistic Data Annotation Specification: Assessment of Fluency and Adequacy in Translations Revision 1.5, January 25, 2005 ,” Tech. Rep., Linguistic
Data Consortium, 2005.
[34] Matthias Sperber, Graham Neubig, Jan Niehues, andAlex Waibel, “Neural lattice-to-sequence models for uncertain inputs,” in Proceedings of the 2017 Conferenceon Empirical Methods in Natural Language Processing,
Copenhagen, Denmark, Sept. 2017, pp. 1380–1389, Association for Computational Linguistics.