Simultaneous Speech-to-Speech
Translation Systemwith Transformer-based
Incremental ASR, MT, and TTS
Ryo Fukuda, Sashi Novitasari, Yui Oka, Yasumasa Kano, Yuki Yano, Yuka Ko, Hirotaka
Tokuyama, Kosuke Doi, Tomoya Yanagita, Sakriani Sakti, Katsuhito Sudoh, Satoshi Nakamura
Nara Institute of Science and Technology (NAIST)Part of this work is supported by JSPS KAKENHI (Grant number JP2H05054).

Copyright © 2021
Katsuhito Sudoh, NAIST

2Video

Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAIST

3Background• Simultaneous Speech-to-Speech Translation from En to Ja• Simultaneous Translation• Not waiting for the end of a sentence; Real-time communication• Speech-to-Speech Translation• No displays for closed captions / subtitles• Simultaneous Translation• No interpretation efforts like summarization• From English to Japanese• Syntactically distant language pair; Large difference in word order
Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAISTCascade Simultaneous S2S Translation System
Input EnglishI am a chef and aspeech

Output Japanesespeech
私はシェフで、⾷food policy guy,
but I come from ...

べ物政策の⼈…

Incremental

Incremental

Incremental

ASR(ISR)

MT(IMT)

TTS(ITTS)Intermediate
English transcript

4

I was a chef and afood policy guy,
but I come from ...

私はシェフで、⾷ Intermediate
べ物政策の⼈… Japanese MT result

Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAIST

5

Incremental Speech Recognition (ISR)• Transformer-based
extension to our ISR
using knowledge
distillation(Novitasari+ 2019)• The student model
mimics the speechtext alignment
</m>TransformerDecoder</m>(vote)
Transformer
Decoder

Enc-dec
multi-headattention

Transformer
Decoder...

Attention Matrix

Transformer
Encoder

Transformer
Encoder

Transformer
Encoder

Step n = 1
Non-incremental ASR

Oriental COCOSDA 2021 (Nov. 20, 2021)

Step n = 2
Incremental ASR

Copyright © 2021
Katsuhito Sudoh, NAIST

6

Incremental Machine Translation (IMT)• Applies our text-to-text IMT(Fukuda et al. 2021)

ブッシュ ⼤統領は今⽇ プーチンとWait-3• Wait-k• Delays the decoding process ink input tokens
President Bush• Sequence-level Knowledge
Distillation• Trains the IMT model using
offline MT results• Japanese chunk shuffling(details in the paper)

meetswithParallel corpus for training

English(source)

JapanesePutintoday.

Simultaneous NMT model(Transformer; wait-k)(target)Pseudo-parallel corpus

Offline NMT model(Transformer)

Oriental COCOSDA 2021 (Nov. 20, 2021)

English(source)

Japanese(MT results)

Copyright © 2021
Katsuhito Sudoh, NAIST

Incremental Text-to-Speech Synthesis (ITTS)• Based on Transformer TTS(Li et al. 2019)• Takes fixed # of Ja words
and convert them into kana(Japanese phonograms)• Predicts Mel-spectrogram• Generates speech signals
using CBHG and Griffin-Lim• Continues until the end-ofsentence

Word sequence
from IMT

TransformerITTS

TransformerITTS

Char-seq 1 ~ 2Char-seq 3 ~ 4Concatenate

Concatenate

Char-seq 1

Char-seq 2Char-seq 3Char-seq 4

Word 1

Word 2

Word 3

Word 4

Step n = 1

Oriental COCOSDA 2021 (Nov. 20, 2021)

7...

Step n = 2

Copyright © 2021
Katsuhito Sudoh, NAIST

8Evaluation Overview• The cascade system was evaluated using TED Talks data in:• System-level latency• Ear-Voice Span (EVS)• Span between the start of the input & output• Module-level quality• ISR: Word Error Rate (WER)• ISR+IMT: BLEU
• ITTS: L2-norm loss and subjective evaluation (AB preference test)• Three latency regimes: low, medium, high (following IWSLT)
Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAIST

9Evaluation: Latency
Latencyregime

LowMediumHigh

ISR delay(sec.)

0.93
0.93
1.30

ISR+IMT delay Total delay (=EVS)(sec.)(sec.)

4.69
8.43
11.47

Hyperparameters

8.81

ISR: 64 frames
IMT: k=10
ITTS: 5 words

11.87

ISR: 64 frames
IMT: k=20
ITTS: 5 words

16.91

ISR: 64 frames
IMT: k=30
ITTS: 7 words

Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAIST

10Evaluation: Quality (ASR & MT)• ISR
Nonincremental• ISR+IMTModel

WER (%)

LSTM

25.46
20.74

Transformer
LSTM(low latency)

LSTM

Incremental(high latency)

Transformer(low latency)

Transformer(high latency)

31.88
32.43
32.06
25.01

Subjective evaluation

BLEU-4(%)

Adequacy

FluencyGoldtranscript+MT15.7

3.41

3.93

Non-incremental
ASR+MT

12.8

3.20

4.01

IMT

5.1

2.80

3.03

IMT

8.4

2.98

3.54

IMT

9.4

3.34

3.80

(low latency)(medium latency)(high latency)

Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAIST

11

Evaluation: Quality (TTS)
Nonincremental

Model

L2-norm loss

Tacotron

0.57
0.51

Transformer
Tacotron(low latency)

Tacotron

Incremental(high latency)

Transformer(low latency)

Transformer(high latency)

0.77

TacotronTTSSame

Transformer TTS

Transformer
ITTS (7 words)

Transformer TTS

Same

0.58
Transformer
ITTS (7 words)

0.65
0.57

0%

25%

Oriental COCOSDA 2021 (Nov. 20, 2021)

Transformer
ITTS (5 words)

50%

Same

75%

100%

Copyright © 2021
Katsuhito Sudoh, NAIST

12Conclusions• Our En-to-Ja simultaneous speech-to-speech translation
system using Transformer-based incremental modules• Module-level delays are still problematic• Suffers from speaking latency for long TTS output• Delay increases when a new output come during speaking• Future work includes:• Aggressive anticipation to reduce the delay• Controlling the speaking duration• Evaluation of simultaneous speech-to-speech translation
Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAIST

13References• Novitasari, S. et al. 2019, Sequence-to-Sequence Learning
via Attention Transfer for Incremental Speech Recognition,
Proc. Interspech 2019
• Fukuda, R. et al., 2021, NAIST English-to-JapaneseSimultaneous Translation System for IWSLT 2021
Simultaneous Text-to-text Task, Proc. IWSLT 2021
• Li, N. et al., 2019, Neural Speech Synthesis with Transformer
Network, Proc. AAAI 2019
Oriental COCOSDA 2021 (Nov. 20, 2021)

Copyright © 2021
Katsuhito Sudoh, NAIST