SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM
WITH TRANSFORMER-BASED INCREMENTAL ASR, MT, AND TTS
Ryo Fukuda, Sashi Novitasari, Yui Oka, Yasumasa Kano, Yuki Yano, Yuka Ko, Hirotaka Tokuyama,
Kosuke Doi, Tomoya Yanagita, Sakriani Sakti, Katsuhito Sudoh, Satoshi Nakamura
Nara Institute of Science and Technology, Japan
ABSTRACTIn this paper, we present an English-to-Japanese simultaneous speech-to-speech translation (S2ST) system. It has
three Transformer-based incremental processing modules for
S2ST: automatic speech recognition (ASR), machine translation (MT), and text-to-speech synthesis (TTS). We also
evaluated its system-level latency in addition to the modulelevel latency and accuracy.
Index Terms— Simultaneous translation, Speech translation, English-to-Japanese translation
1. INTRODUCTION
Speech-to-speech translation (S2ST) is a promising and challenging technology for assisting cross-lingual human conversation [1, 2, 3]. Recent deep learning technologies advanced
speech and language processing, and many studies addressed
the problem of real-time automatic S2ST. However, we face acrucial problem of the delay of the S2ST processes. Since an
S2ST system usually handle speech inputs at the utterance or
sentence level, it has to wait for the end of an utterance so thatthe delay becomes proportional to the length of the input. It is
not useful for long monologues such as lectures. Simultaneous interpretaion is often used in such situations, but it is alsoa very challenging task that requires experienced interpreters.
In this paper, we focus on the problem of simultaneous
S2ST from English to Japanese and present a system basedon neural networks called Transformer. Note that we differentiate simultaneous translation from simultaneous interpretation, because the current simultaneous translation does
not include interpretation efforts such as summarization. This
problem requires real-time and incremental processing thatworks simultaneously with the input. Most previous attempts
for simultaneous speech translation focused on speech-to-texttranslation between English and Europearn languages [4, 5,
6]. Our work aims for S2ST from English to Japanese. English and Japanese are very different in their syntax and difficult to translate each other. Our system cascades three modules: incremental speech recognition (ISR), incremental machine translation (IMT), and text-to-speech synthesis (ITTS).
End-to-end approaches are used in recent studies on speechtranslation [7], but it is still difficult to apply them for Englishto-Japanese S2ST [8].
We evaluate our system in system-level latency in addition to module-level performance on S2ST from English TED
Talks to Japanese. We have two system-level latency metrics:(1) the system-level Ear-Voice Span consisting of computation time and cascading delay, and (2) the cumulative speaking latency derived from the overlap of TTS outputs. The ISR,
IMT, and ITTS modules are evaluated by their standard metrics. This is the first attempt of a system-level evaluation fora simultaneous S2ST system and will benefit future studies.
2. SIMULTANEOUS SPEECH-TO-SPEECH
TRANSLATION
Given input sequence X = x1 , x2 , ..., x|X| , we predictthe corresponding output sequence Y = y1 , y2 , ..., y|Y |by sequence-to-sequence transduction. In simultaneoustranslation, we make the prediction one-by-one on subsequences. Suppose we have predicted output subsequence
Y1j = y1 , y2 , ..., yj from partial input observations X1i =0ix1 , x2 , ..., xi . When we observe the next partial input Xi+1=
xi+1 , ..., xi0 , we predict the corresponding output subse0

jquence Yj+1
= yj+1 , ..., kj 0 based on the following formula:
0

0

j
i
Yj+1
= argmax P (Ŷ |X1i , Xi+1
, Y1j ),

(1)Ŷ

where Ŷ is a subsequence-level partial prediction.
Here we face a problem in deciding the length of a partial
observation to make a partial prediction. Non-simultaneoustranslation waits until the end of the sentence, but we must
make partial predictions incrementally for simultaneoustranslation. This work uses simple fixed-length criteria forincremental processing, as described in Section 3.
3. INCREMENTAL PROCESSING MODULES
3.1. Incremental speech recognition
We tackle the latency problem in ASR by using our ISR
method [9] on the Transformer-based encoder-decoder model




</m>

Transformer
Decoder</m>

Transformer
Decoder

Enc-dec
multi-headattention

Transformer
Decoder...

Attention Matrix

Transformer
Encoder

TransformerITTS

TransformerITTS

Char-seq 1 ~ 2Char-seq 3 ~ 4Concatenate

Concatenate(vote)


Transformer
Encoder

Step n = 1
Non-incremental ASR...

Transformer
Encoder

Char-seq 1

Char-seq 2Char-seq 3Char-seq 4

Word 1

Word 2

Word 3

Word 4

Step n = 2
Incremental ASR

Fig. 1. Transformer-based ISR construction with attention
transfer [9] from a standard Transformer-based ASR

Word sequence
from IMT

Step n = 1

Step n = 2

Fig. 2. Incremental text-to-speech synthesis system[10]. We use a teacher-student framework to train the ISR
model. We firstly train a non-incremental ASR as the teacher
model and then train an ISR as a student model to do blockwise input processing by learning the knowledge of theteacher model. The student ISR model has the same model architecture as the teacher ASR model and learns to mimic the
speech-text alignment based on the encoder-decoder attention
by the teacher model.
Suppose the teacher ASR model transcribes speech utterance X = x1 , x2 , ..., x|X| with length |X| into token sequence S = s1 , s2 , ..., s|S| with length |S|. The teacher’s
attention-based alignment is extracted from the attention between the hidden states of the encoder and decoder. From theattention sequences in all the layers and heads, the speech-textalignment for the ISR training was generated through teacherforcing decoding to decide the final alignment A by a majority vote. We split X into M sub-segments X̄ = [x̄1 , . . . , x̄M ]and S into S̄ = [s̄0 , . . . , s̄M ], using A. Here, an end-of-block
symbol </m> is added at the end of each sub-segment.
ISR learns the incremental steps by learning a pair of X̄
(with the length of the W frames for each speech segment)
and S̄ (Fig. 1). By inference, ISR performs an incremental
recognition step when the speech buffer achieved W frames
to predict the corresponding transcription and then moves tothe next incremental step when it predicts a </m> symbol.
In our ISR, we allowed the model to take look-back and lookahead input sequences as the contextual input to provide moredetailed speech information.
3.2. Incremental machine translation
MT suffers from a reordering problem, caused by the syntactic differences between the source and target languages.
Although it is not serious in current neural MT [11, 12], it
remains very problematic when we consider IMT because wecannot observe a complete sentence. Previous studies took
inputs and predicted the corresponding outputs incrementallyin an adaptive manner [13, 14]. In this work, we use a method
called wait-k, waits for the first k input tokens before starting
the decoding process [15]. The wait-k IMT model translatesa token sequence of source language S = s1 , . . . , s|S| into
target language T = t1 , . . . , t|T | . The decoder predicts an
output token using a partial input. k is a hyperparameter for
the fixed number of input tokens for the initial wait; setting klarger leads larger delays, and a smaller k worsens the output
predictions due to poor context information.
In this work, we also introduce two extensions of IMT
training: knowledge distillation and chunk shuffling. We distill knowledge from a non-incremental teacher MT model toa student wait-k model by sequence-level knowledge distillation [16]. We apply chunk shuffling, random reorderingof Japanese chunks called bunsetsu, for data augmentation.
We expect it encourages the IMT to be monotonic, based ona characteristic of Japanese in which the order of bunsetsu
chunks can be relaxed. Suppose we have target language sentence T = t1 , . . . , t|T | in our training set. We segment T
greedily into a sequence of chunks T̄ = C1 , . . . , CQ . The
length of each chunk is set to k (i.e., delay hyperparameter in
wait-k) tokens: Cq = tq1 , . . . , tqk , except for the last chunk
CQ that can be shorter than k. Finally, we shuffle the chunkswith a pre-defined fixed probability pr .
3.3. Incremental text-to-speech synthesis
ITTS synthesizes speech based on a short text segment. In
this work, our ITTS structure is based on the Transformerbased TTS proposed by Li et al. [17]. It consists of an encoder and a decoder, both of which have a Transformer structure. Given the input sequence of tokens, the encoder maps
the input into semantic space to generate the sequence of theencoder’s hidden states. This sequence is then utilized by
the decoder, along with the decoder output in the previous
timestep, to predict the speech’s Mel-spectrogram and a stoptoken that marks the end-of-sentence. In this work, from theMel-spectrogram, we generated speech signals by generatinga magnitude spectrogram using a CBHG (1-D Convolution
Bank + Highway + bidirectional GRU) module that resembles the Tacotron framework [18], followed by speech phasespectrogram estimation with the Griffin-Lim algorithm and an
inverse short-time Fourier transform (STFT).
The process of our ITTS module in a S2ST system is illustrated in Fig. 2. For each incremental step, ITTS takes thecharacter sequence of a fixed number of words and synthesizes the corresponding speech. Since it takes a sequence ofJapanese kana phonograms as input, the output of the preceding IMT, including Japanese kanji morphograms, must be
converted accordingly. We applied lattice-based tokenizationand kanji-to-kana conversion1 to IMT output strings, whichwere de-tokenized from subword sequences.
We trained the ITTS using word-level alignment between
speech and text2 . Given complete text T = [t1 , t2 , ..., t|T | ]with length |T | and corresponding speech utterance Y =[y1 , y2 , ..., y|Y | ] with length |Y |, we split T into Q subsegments T̄ = [t̄0 , . . . , t̄Q ] and Y into Q sub-segmentsȲ = [ȳ0 , . . . , s̄Q ], based on the alignment. All T̄ subsegments have the same number of words, which are converted into characters when training the ITTS. Each Ȳ subsegment is concatenated with a blank at the end to mark theend-of-speech segment. Here, the blank tensor corresponds
to a stop token label during training. By inference, ITTS
stops the incremental step when the decoder predicts the stoptoken and proceeds to the next incremental step by taking thenext step’s input. We allowed the model to take look-backand look-ahead input sequences as contextual input. Similar
to the main input, the number of words in the contextual input
is fixed and also learned during the ITTS training.
4. EVALUATION
We evaluated our system by an English-to-Japanese simultaneous S2ST experiment. We investigated the detailed system
performance by focusing on: (1) system-level latency and (2)module-level quality.
4.1. Evaluation setup
The ISR, IMT, and ITTS models were trained with a few
different latency parameters for comparison, motivated by
IWSLT evaluation campaign [20].
We constructed ISR with Transformer-big-based model
configuration proposed in Speech-Transformer [10]. The ISR
training was done using TED-LIUM release 1 [21] with a
total of 774 talks (56.8k cut utterances, representing about
118 hours of speech). We extracted 80 dimensions of Melspectrogram features with a 50-ms frame window and a 12.5ms window shift as the ISR input. The English speech transcriptions in the ISR training were segmented into subwords
using a byte-pair encoding3 model, which is identical as the
1 We used PyKakasi for the tokenization and morphogram conversion(https://github.com/miurahr/pykakasi)
2 We generated the alignment using Montreal Forced Aligner [19]3 subword-nmt toolkit (https://github.com/rsennrich/subword-nmt)

one in the IMT part. We trained two ISR systems with different input delay configurations: 64 frames and 96 frames.
The ISR with 64 frames of input required 32 frames as themain input and the next 32 frames as the look-ahead input foreach incremental step. In the ISR with 96 frames, it takes 64
frames as the main input and 32 look-ahead frames for eachincremental step. Both models are also allowed to see thelook-back frames with a range of 256 frames. The modulelevel ISR systems were evaluated on the TED-LIUM release
1 test set (“TED-LIUM test”) that consisted of 1155 cut shortspeech utterances (average 7.88 sec) of long TED talks.
The IMT model configuration was based on Transformerbase [12]. We first trained the model using the WMT 2020
news task data (17.9 million sentence pairs) and fine-tuned
using IWSLT 2017 data (223 thousand sentence pairs). In thefine-tuning, we examined different configurations for knowledge distillation and chunk shuffling and chose the configuration based on the results on the validation data. We tokenized the sentences into subwords based on Byte Pair Encoding [22]. The vocabulary was shared over the source andtarget languages using 16,000 entries.
The ITTS model configuration was based on Transformerbased TTS [17]. We utilized the JSUT dataset [23] for modeltraining, which consists of one Japanese woman’s speech with
5.2k pairs of speech utterances and their corresponding transcriptions. We used 5k utterances for training, 100 for development, and 100 for test. We extracted 80 Mel-spectrogram
dimensions with a 50-ms frame window and a 12.5-ms window shift as the speech feature targets. We constructed two
ITTSs with different input delays in an incremental step: 5
words and 7 words. The ITTS with input of 5 words took
3 main words and 2 look-ahead words for each incremental
step. The ITTS with input of 7 words took 5 main words and2 look-ahead words as the input for each incremental step.
Both models were also allowed to see the look-back input inthe range of 10 words.
The system-level test set consisted of eight TED talks(1.89 hours in total) with English and Japanese subtitles.
4.2. Latency
We measured the latency of our system with two different
metrics: Ear-Voice Span (EVS) and cumulative speaking latency.
4.2.1. Ear-Voice Span (EVS)
EVS is a common measure of simultaneous interpretation latency, calculated as the delay between the start of input andinterpretation speech. We used it for the evaluation of thesystem-level delay, which we call system-level EVS, causedby the module-level processing and inter-module communications. For this purpose, we aligned the module-level outputs
with the output timestamps (Fig. 3). The delays are shown in
Table 1.

Time (sec) 0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Small Latency

and a food policy guy| i was a chef

ISR
IMT
ITTS

but i come from a whole family teacher 

| watashi wa shefu de tabemono seisaku no hito deshita ga watashi ha kyoushi no kazoku zeiin kara kimashita 

| tabemono seisaku no hito deshita ga watashi wa kyoushi no| watashi hakazoku zeiin kara kimashita 
shefu de
IMTdelay

ISR delay

ITTS delay (start speaking)

ITTS finish speaking

Medium Latency| i was a chef

ISR
IMT

and a food policy guy

but i come from a whole family teacher 

| watashi ha shefu de ari tabe mono seisaku no otoko deshita ga watashi wa kazoku no kyousi kara
kimashita 

IMTdelay

ITTS

ISR delay| watashi wa shefu de ari tabemono seisaku no otoko deshita ga watashi
wa kazoku no kyousi kara kimashita 


ITTS delay (start speaking)

ITTS finish speaking

High Latency| i was a chef

ISR
IMT

and a food policy guy

ISR delay

but i come from a whole family of teachers 

| watashi wa shefu de tabemono seisaku no otoko deshita ga watashi wa kyoushi no kazoku zeiin kara
kimashita 

IMT delay

ITTS| watashi wa shefu de tabemono seisaku no otoko deshita ga watashi wa kyoushi no kazoku
zeiin kara kimashita 


ITTS delay (start speaking)

ITTS finish speaking

Fig. 3. ISR, IMT, and ITTS output and alignment example
SystemLow latency
Medium latency
High latency

ISR delay
0.93(64 frames)
0.93(64 frames)
1.30(96 frames)

IMT delay
4.69(k 10)
8.43(k 20 )
11.47
(k 30)

ITTS delay
8.81(5 words)
11.87(5 words)
16.91(7 words)

Table 1. Module-level processing delays (sec) from beginning of speech inputs by ISR, IMT, and ITTS: Experiment
was conducted on long talk source speech (average length14.15 min). Note: later modules have to work after their preceding module, so they include delays in preceding module.
From Table 1, we can see that the ISR worked with fixed
delays, while the latter IMT and ITTS modules were influenced by the delays of the preceding modules.
4.2.2. Cumulative speaking latency
We identified long delays by the ITTS module. A TED Talk
speaker generally talks smoothly without disfuency and longpauses, so following inputs come to the ITTS module evenduring the play of the previous output waveform (Fig. 3). As
a result, such TTS outputs were queued and played later withlarge delays, which we call the cumulative speaking latency.
This overlap becomes critical in the long speech inputs. In
this work, we conducted segment-based evaluation and will
tackle talk-level evaluation in future studies.
4.3. Quality
We evaluated the quality of the results by our system, including the module-level one for all three modules: ISR, IMT, andITTS. In the subjective evaluation described below, twelve
graduate students evaluated the results of IMT and ITTS.

TED-LIUM test
(Cut utterances)
Topline (non-incremental)
Seq2seq LSTM
23.78
Seq2seq Transformer
21.15
Baseline: Seq2seq ISR (LSTM)
Input = 64 frames/step
29.01
Input = 96 frames/step
28.80
Proposed: Transformer ISR
Input = 64 frames/step
28.55
Input = 96 frames/step
27.31
Model

TED talk test
(Long talks)
25.46
20.74
31.88
32.43
32.06
25.01

Table 2. ASR performance (WER%) on TED-LIUM and our
TED test data.
4.3.1. Incremental speech recognition
We evaluated the ISR results with the word error rate (WER).
We compared our ISR with the seq2seq LSTM-based ISR
with attention transfer [9] as the baseline and the standard
non-incremental ASR as our topline (Table 2). Our proposed
Transformer-based ISR outperformed the LSTM-based ISR,
especially in the long talk recognition in which the task reflects the speech condition in simultaneous S2ST.
4.3.2. Incremental machine translation
We evaluated the ISR+IMT results with BLEU [24] using
TED Japanese subtitle as the reference, calculated by SacreBLEU. Table 3 shows the results. Unfortunately, our BLEU-4
result was very low for the following various reasons: (1) Domain mismatch: the in-domain (TED) training data was notlarge; (2) Style mismatch: the TED Japanese subtitles are often idiomatic translations and they may not be suitable forsurface-based evaluation like BLEU; and (3) ASR error propagation.

ASR
MT
WER BLEU
ST Topline (non-incremental)
Correct text + MT
0.00
15.7
Standard ASR + MT
20.74
12.8
ST with baseline IMT (incremental)
ISR (64) + IMT (small)
32.06
4.5
ISR (64) + IMT (medium) 32.06
7.5
ISR (96) + IMT (high)
25.01
8.1
ST with proposed IMT (incremental)
ISR (64) + IMT (small)
32.06
5.1
ISR (64) + IMT (medium) 32.06
8.4
ISR (96) + IMT (high)
25.01
9.4
SystemSubjective Evaluation
Adequacy Fluency
3.41
3.20

3.93
4.01

2.60
2.86
3.31

2.56
3.30
3.82

2.80
2.98
3.34

3.03
3.54
3.80

Tacotron
TTS

Transformer
ITTS (7 words)

Transformer TTS

Transformer
ITTS (7 words)

0%

Same

Transformer TTS

25%

Transformer
ITTS (5 words)

50%

Same

Same

75%

100%

Fig. 4. AB preference test scores for ITTS on TED test data.

Table 3. Speech translation performance on our TED talk test
data.
4.4. Discussion

System
TTS (non-incremental)
ITTS (5 words/step)
ITTS (7 words/step)

Tacotron
TTS
0.57
0.77
0.65

Transformer
TTS
0.51
0.58
0.57

Table 4. L2-norm loss between Mel-spectrogram of originaland ITTS speech on JSUT test data.

We also subjectively evaluated the ISR+IMT results,
based on 1-5 scale adequacy and fluency metrics used in past
MT evaluation campaigns [25]. The rightmost columns inTable 3 show the results. The adequacy and fluency results
showed correlation with the BLEU-1 results, which are dominated by word unigram precision. The proposed method
demonstrated better results than the baseline in the small andmedium latency conditions.

4.3.3. Incremental speech synthesis
Our ITTS evaluations were done with objective and subjective evaluations. We conducted an objective evaluation on
JSUT using L2 loss between the correct Mel-spectrogram andthe Mel-spectrogram predicted by ITTS (Table 4). Here ourbaselines are the ITTS with a Tacotron (LSTM) structure, andthe topline is the standard non-incremental Transformer TTS.
For the subjective evaluation, we performed an AB preference test with our TED talk test data to compare and evaluate the naturalness of the synthesized speech. Sixty pairs ofsynthetic speech samples generated by using different methods were presented to listeners in random orders. The results
show that the performance of the Transformer-based system
was better than the Tacotron-based system. With respect tothe difference between delays in 5 and 7 words, there were no
remarkable differences both in L2-norm and subjective evaluation. Thus, the ITTS worked efficiently using 5-word delay
with comparable quality to that using 7-word delay.

From the latency viewpoint, our cascade simultaneous S2ST
system worked successfully with relatively short delays.
However, the problem is still challenging in quality due
to various reasons including the error propagation by the cascade and data scarcity. Tight integration of the modules suchas a lattice-to-sequence [26] is promising, although it is nottrivial to apply such integration into simultaneous translation.
Unfortunately, we still do not have common system-levelevaluation methodologies and metrics for simultaneous S2ST
other than module-level ones. The two metrics used in thiswork focused only on latency, so we need to evaluate content
delivery through objective measurement. Comparison withhuman interpreters in terms of content delivery and user satisfaction are very important in future.

5. CONCLUSIONSWe presented our English-to-Japanese simultaneous S2ST
system and evaluated it using a TED talks dataset. The system
works incrementally by the cascaded incremental processingof ASR, MT, and TTS, implemented based on Transformer. A
latency evaluation revealed that module-level delay remains
problematic in incremental MT and TTS, even though it canbe controlled by delay hyperparameters at the cost of a dropin accuracy. Our speech-to-speech simultaneous translation
system also suffers from speaking latency.
In future work, we aim to improve the accuracy and efficiency of the modules based on aggressive anticipation usinglarge-scale pre-trained models and to decrease the ITTS latency by controlling the speaking duration.

6. ACKNOWLEDGMENTS
Part of this work was supported by JSPS KAKENHI Grant
Number JP21H05054.

7. REFERENCES[1] Satoshi Nakamura, Konstantin Markov, Hiromi
Nakaiwa, Genichiro Kikui, Hisashi Kawai, Takatoshi
Jitsuhiro, Jin-Song Zhang, Hirofumi Yamamoto, Eiichiro Sumita, and Seiichi Yamamoto, “The ATR
Multilingual Speech-to-Speech Translation System,”IEEE Trans. ASLP, vol. 14, no. 2, pp. 365–376, 2006.
[2] Eiichiro Sumita, Tohru Shimizu, and Satoshi Nakamura,“NICT-ATR speech-to-speech translation system,” inProc. ACL: Demo, 2007, pp. 25–28.
[3] Sakriani Sakti, Michael Paul, Andrew Finch, Xinhui Hu,
Jinfu Ni, Noriyuki Kimura, Shigeki Matsuda, ChioriHori, Yutaka Ashikari, Hisashi Kawai, Hideki Kashioka, Eiichiro Sumita, and Satoshi Nakamura, “Distributed speech translation technologies for multiparty
multilingual communication,” ACM Trans. SLP, vol. 9,
no. 2, Aug. 2012.
[4] Jan Niehues, Ngoc-Quan Pham, Thanh-Le Ha, Matthias
Sperber, and Alex Waibel, “Low-latency neural speechtranslation,” in Proc. Interspeech, 2018, pp. 1293–1297.
[5] Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin,
Zhou Zhao, and Tie-Yan Liu, “SimulSpeech: End-toend simultaneous speech to text translation,” in Proc.
ACL, 2020, pp. 3787–3796.
[6] Xutai Ma, Juan Pino, and Philipp Koehn, “SimulMT
to SimulST: Adapting simultaneous text translation to
end-to-end simultaneous speech translation,” in Proc.
AACL-IJCNLP, 2020, pp. 582–587.
[7] Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey,
Melvin Johnson, Zhifeng Chen, and Yonghui Wu, “Direct speech-to-speech translation with a sequence-tosequence model,” in Proc. Interspeech, 2019, pp. 1123–
1127.
[8] Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura,“Transformer-Based Direct Speech-To-Speech Translation with Transcoder,” in Proc. IEEE SLT, 2021, pp.
958–965.

[12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser,
and Illia Polosukhin, “Attention is All you Need,” inProc. NIPS, pp. 5998–6008. 2017.
[13] Tomoki Fujita, Graham Neubig, Sakriani Sakti, Tomoki
Toda, and Satoshi Nakamura, “Simple, Lexicalized
Choice of Translation Timing for Simultaneous Speech
Translation,” in Proc. Interspeech, 2013, pp. 3487–
3491.
[14] Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li, “Learning to translate in real-time withneural machine translation,” in Proc. EACL, 2017, pp.
1053–1062.
[15] Mingbo Ma, Liang Huang, Hao Xiong, Renjie
Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang,
Zhongjun He, Hairong Liu, Xing Li, Hua Wu, andHaifeng Wang, “STACL: Simultaneous translation
with implicit anticipation and controllable latency using
prefix-to-prefix framework,” in Proc. ACL, 2019, pp.
3025–3036.
[16] Yoon Kim and Alexander M Rush, “Sequence-levelknowledge distillation,” arXiv preprint 1606.07947,
2016.
[17] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, andMing Liu, “Neural speech synthesis with transformer
network,” in Proc. AAAI, 2019, pp. 6706–6713.
[18] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui
Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying
Xiao, Zhifeng Chen, Samy Bengio, et al., “Tacotron:
Towards end-to-end speech synthesis,” in Proc. Interspeech, 2017, pp. 4006–4010.
[19] Michael McAuliffe, Michaela Socolof, Sarah Mihuc,
Michael Wagner, and Morgan Sonderegger, “Montreal
Forced Aligner: Trainable text-speech alignment using
Kaldi,” in Proc. INTERSPEECH, 2017, pp. 498–502.

[9] Sashi Novitasari, Andros Tjandra, Sakriani Sakti, andSatoshi Nakamura, “Sequence-to-Sequence Learning
via Attention Transfer for Incremental Speech Recognition,” in Proc. Interspeech, 2019, pp. 3835–3839.

[20] Antonios Anastasopoulos, Ondřej Bojar, Jacob Bremerman, Roldano Cattoni, Maha Elbayad, MarcelloFederico, Xutai Ma, Satoshi Nakamura, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Sebastian Stüker, Katsuhito Sudoh, Marco Turchi, Alexander
Waibel, Changhan Wang, and Matthew Wiesner, “Findings of the IWSLT 2021 evaluation campaign,” in Proc.
IWSLT, 2021, pp. 1–29.

[10] L. Dong, S. Xu, and B. Xu, “Speech-Transformer: A
no-recurrence sequence-to-sequence model for speech
recognition,” in Proc. ICASSP, 2018, pp. 5884–5888.

[21] Anthony Rousseau, Paul Deléglise, and Yannick Estève,“TED-LIUM: an automatic speech recognition dedicated corpus,” in Proc. LREC, 2012, pp. 125–129.

[11] Thang Luong, Hieu Pham, and Christopher D. Manning,“Effective approaches to attention-based neural machinetranslation,” in Proc. EMNLP, 2015, pp. 1412–1421.

[22] Rico Sennrich, Barry Haddow, and Alexandra Birch,“Neural machine translation of rare words with subword
units,” in Proc. ACL, 2016, pp. 1715–1725.

[23] Ryosuke Sonobe, Shinnosuke Takamichi, and Hiroshi
Saruwatari, “JSUT corpus: free large-scale Japanese
speech corpus for end-to-end speech synthesis,” arXiv
preprint 1711.00354, 2017.
[24] Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu, “Bleu: a method for automatic evaluation ofmachine translation,” in Proc. ACL, 2002, pp. 311–318.
[25] LDC, “ Linguistic Data Annotation Specification: Assessment of Fluency and Adequacy in Translations Revision 1.5, January 25, 2005 ,” Tech. Rep., Linguistic
Data Consortium, 2005.
[26] Matthias Sperber, Graham Neubig, Jan Niehues, andAlex Waibel, “Neural lattice-to-sequence models for uncertain inputs,” in Proc. EMNLP, 2017, pp. 1380–1389.