1

Recent Advances in Speech-to-speech
Translation
Dr. Satoshi Nakamura*with Katsuhito Sudo, Sakriani Sakti# ,and Ryo Fukuda, Sashi Novitasari, Tomoya Yanagita,
Kosuke Doi, Yasumasa Kano, Yuka Ko, Yuki Yano, Hirotaka Tokuyama, Yui Oka*Director, Data Science Center,

Professor, Graduate School of Science and Technology,

Nara Institute of Science and Technology(#currently with JAIST)
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

2

Talk Outline
Recent advances– Machine translation (text-to-text)– Speech recognition (Speech-to-text)– Speech synthesis (Text-to-speech)
Speech translation– Speech translation research history– Simultaneous speech translation

Summary and future directions

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

3

Recent MT progress
Rule-based MT：–

Linguists generate translation rules

Corpus-based MT:–––

Example-Based: Automatic rule extraction from corpus[M.Nagao84, Sato et.al.,89, Sumita et. al., 91 ]Statistical MT: Statistical Modeling of MT based on Noisy Channel Model [P.F.Brown, et.al. 93]
Phrase-base SMT•

P.Koehn, et al., “Statistical Phrase-based Translation”, UNIVERSITY OF SOUTHERN CALIFORNIA MARINA DEL REY INFORMATIONSCIENCES INST 2003

Tree-to-string–

Statistical MT based on Tree Structure•

Y.Liu, et al,” Adjoining Tree-to-String Translation”, Proc. ACL 2011

Neural Machine Translation–

Combination of Encoder and Decoder by LSTM•

I.Sutskever, “Sequence to Sequence Learning with Neural Networks”, Proc. NIPS 2014

Attention NMT–

Add Attention to encoder and decoder•

D.Bardanau, et al., “Neural Machine Translation by Jointly Learning to Align and Translate”, Proc. of ICLR 2014

Transformer NMT–

Self attention by multiple heads. Transformer.
•

Ashish Vaswani et al., Attention Is All You Need, Proc. of NIPS 2017.

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

4

Phrase Based Statistical Machine Translation
Divide the sentence into small phrases and translate⚫

Today I will give a lecture on machine translation .
Today
I will give
今日は、 を行います

Today
今日は、

a lecture onの講義

machine translation
機械翻訳

machine translation
機械翻訳

a lecture onの講義

I will give
を行います

.
。

.
。

今日は、機械翻訳の講義を行います。
kyowa kikaihonyaku no kogi wo okonaimasu⚫

Score translations with translation model (TM),reordering model (RM), and language model (LM)

無限の可能性、ここが最先端 －Outgrow your limits－©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021
Nov. 12 2021

http://www.naist.jp/

5

Statistical MT• Translation model, Language Model, Distortion Model
Source and target
language paralleltext corpus

Target language
text corpus

Parameter estimation

Translation modelParameter estimation

Reordering modelPhrase substitution
Input text(
Source Language)

無限の可能性、ここが最先端 －Outgrow your limits－

Language model
Grammaticalcorrectness

Machine Translation
Decoding

Nov. 12 2021

Translation text
(Target Language)©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

6

Encoder-decoder Model [Sutskever+ 14]
Memorize input sentence by LSTM recurrent neural network
Generate output sentence by LSTM recurrent neural network
Memorize
Sentence
これ
kore

は 機械
翻訳
です
wa kikai honnyaku desu

Generate MT sentence
looking back the memory

Vector Representation

Decoder

Encoder

Vector Representation

Memorize input
sentence

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

This

is

a

machine translation©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

7

Attention Mechanism [Bahdanau+ 14]
Better Memorization of Sentence and Looking-back Mechanism– Weighted-sum by the attention

これ
kore

は 機械
翻訳
です
wa kikai honnyaku desu

Looking back byAttention Mechanism

Vector Representation

Vector RepresentationThis

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

isamachine translation©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

Transformer: Fully Attention-based NN [Vaswani+ 2017]o No RNNs, stacked NNs Self-attention to extract
context dependent info.
o Positional encoding
instead of recurrent stepso Multi-head attention toutilize various aspects

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

8

9

Talk Outline
Recent advances– Machine translation (text-to-text)– Speech recognition (Speech-to-text)– Speech synthesis (Text-to-speech)
Speech translation– Speech translation research history– End-to-end speech translation– Simultaneous speech translation

Summary and future directions
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

10

Recent Progress of ASR
Traditional Technologies– Template Matching, Dynamic Programing [Sakoe 71]– Hidden Markov Modeling, N-Gram Model [Mercer 83, etc]
– Neural Network, TDNN[Waibel 89], LSTM [Hochreiter 97]– Weighted Finite State Transducer [Mohri 2006]– Big Training Data, Data Collection through Trial Service

Deep Learning– DNN-HMM [Hinton 2012]• Estimate State Posterior Probability by DNN– Connectionist Temporal Classification[Graves 2013]• Predict Phoneme Label every frame– Listen, Attend, and Spell [Chan 2016]• CTC+Attention: End-to-end modeling– Transformer ASR• Faster calculation by Multiple-head attention
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

11

Speech Recognition
1960

1990

Template matching
dynamic time warping

Statistical-based HMM

2014+
End-to-end ASR

Speech B

Speech A

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

12

Recent Performance
Saon, et al. “English Conversational Telephone Speech Recognition
by Humans and Machines”, INTERSPEECH 2017

[1] R. P. Lippmann, “Speech recognition by machines and humans,”
Speech communication, vol. 22, no. 1, pp. 1–15, 1997.

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

13

Talk Outline
Recent advances– Machine translation (text-to-text)– Speech recognition (Speech-to-text)– Speech synthesis (Text-to-speech)
Speech translation– Speech translation research history– Simultaneous speech translation

Summary and future directions

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

14

Recent Speech Synthesis
Formant-based Synthesis, Waveform Concatenation

Statistical Speech Synthesis: HTS– Speech Synthesis by HMM•

Tokuda, et al., “Speech parameter generation algorithms for HMM-based speech synthesis”, ICASSP 2000

WaveNet– Waveform Convolution• van den Oord et al., “WAVENET: A GENERATIVE MODEL FOR RAW AUDIO”, arXiv:1609.03499v2 [cs.SD] 19 Sep 2016

Tacotron (Encoder-decoder DNN TTS+ Griffin Lim)
– End-to-end speech synthesis with character input. Waveform generation by Griffin-Lim• Wang, et al., “TACOTRON: TOWARDS END-TO-END SPEECH SYNTHESIS”, arXiv:1703.10135v2 [cs.CL] 6 Apr 2017

Tacotron2 (Encoder-decoder DNN TTS+ Wavenet)• J. Sheng, et al, “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions”,

Transformer TTS:
– Multi-head attention• N. Li, S. Liu, Y. Liu, S. Zhao, and M. Liu, “Neural speech synthesis with Transformer network,” in Proc. AAAI Conference on
Artificial Intelligence, vol. 33, no. 01, 2019, pp. 6706–6713• M. Chen, X. Tan, Y. Ren, J. Xu, H. Sun, S. Zhao, and T. Qin,“MultiSpeech: Multi-speaker text to speech with Transformer,”in Proc. INTERSPEECH, 2020, pp. 4024–4028.
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

15

Speech Synthesis
Conditional WaveNet – TTS

HMM

Tacotron[Oord et al., 2016][Zen et al. 2009]
無限の可能性、ここが最先端 －Outgrow your limits－[Wang et al.; 2017]
Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

History of Speech Translation Research in Japan
1986

1992
ATR

NAIST

1999

ATR

2006

Daily
Conversation• Syntactically correct• Clear utterance• Limited domain
Ex. “Conference
Registration”• Standard expression• Unclear utterance• Limited domain
Ex. “Hotel Reservation”

2010

NICT

ATR

Read
Speech

2008

16

Wider andReal Domain+ More languages
for translation•21 multilateral text translation
VoiceTra• Wider and real domain“International Travel”
A-STAR• Realistic expressions• Noisy speech• J-E, J-C speech translation • Multilateral translation for 8 Asian languages• Network-based S2ST

2011

C-STAR

IWSLT• Multilateral translation for 7 world languages

Fundamentals Rule-based

Statistical
MachineTranslation

MachineTranslation
無限の可能性、ここが最先端 －Outgrow your limits－• Evaluation Campaign of S2S technologies

Nov. 12 2021

NeuralMachineTranslation©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

17

Simultaneous Interpretation
European Commission
Simultaneous interpreting is a mode of interpreting in which the speaker makes aspeech and the interpreter reformulates the speech into a language his audience
understands at the same time (or simultaneously).
The three main actions are also essentially the same as consecutive interpreting.
1) listen actively (understand)2) analyse (structure the message)3) reproduce (communicate)
The difference with consecutive interpreting is that in simultaneous interpreting
all of these things need to happen at the same time (or simultaneously).
In addition to a special way of listening, prioritising information anddistinguishing between primary and secondary information, activating shortterm memory, communicating, etc. ,a good simultaneous interpreter also has to be able to anticipate what the
speaker might say.
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

18

Problem: Delay (Ear-Voice Span)

Delay
ASR
こんにちは、駅はどこですか？
konnichiwa eki wa dokodesuka
MT
Hello, where is the station?
TTS

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

19

Simultaneous Incremental Speech Translation
Delay: Reduced
ASR
こんにちは、駅はkonnichiwa ekiwa

MT

どこですか？
dokodesuka

MTMT

Hello,

the station where is it?

TTS

TTS

TTS

But, this is not easy!
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

20

Cascade Simultaneous S2S Translation System

I am a chef and a foodpolicy guy,
but I come from ...

Input English
speech

Output Japanese
speech

私はシェフで、食べ物政策の人…

Incremental

Incremental

Incremental

ASR
(ISR)

MT
(IMT)

TTS
(ITTS)

Intermediate
English transcript

I was a chef and a foodpolicy guy,
but I come from ...

私はシェフで、食べ物 Intermediate
Japanese MT result
政策の人…

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India
無限の可能性、ここが最先端 －Outgrow your limits－
Nov. 12 2021

http://www.naist.jp/

STACL: Wait-k Algorithm [M.Ma, et al., 2018]

Encoder-decoder NN

ブッシュ

大統領

は

今日

with

Putin

today

.

プーチン

21

と

Wait-3President

Bush

meets

Mingbo Ma, et al., “STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency”, arXiv:1810.08398v3 [cs.CL] 3 Nov 2018
無限の可能性、ここが最先端 －Outgrow your limits－©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India
Nov. 12 2021

http://www.naist.jp/

22Translation Timing Control by Syntactic Prediction in SMT (2015,2021)
Syntax Prediction Process

Syntactic Prediction [Oda, et al., 2015]
1. Parse the input as-is– Incremental bottom up parsing– Feature extraction and syntactic prediction

PP
IN

Subject Verb Object : English
Subject Object Verb: Japanese

DT

JJ

CD

NP
NNS

in the next 18 minutes
Input translation unit

Wait MT output when specific labels appear.

Use LSTM and BERT to predict next tag.
[Kano, et al., 2021]

Word:R1=I
POS:R1=NN
Word:R1-2=I,minutes
POS:R1-2=NN,NNS
...

NP
NP

– Control MT output timing according
reordering

2. Extract features

10/16/2016

NN

I

ROOT=PPROOT-L=IN
ROOT-R=NP...

3. Predict the next tag(linear SVM)
VP ... 0.65
NP ... 0.28
4. Append tonil ... 0.04
sequence
...

VP

nil

5. Repeat until nil‘m going to take you on a journey
46

Invited Talk ©Satoshi Nakamura, NAIST

Increment
al parsing
andsyntactic
predictionin the next 18 minutes[NP]predict [VP] (wait)

MT results

18 分 で あ る[NP]i 'm going to take [keep]
i 'm going to take you on a journey [VP end]

[VP](wait) を 行 っ て い ま す[keep]

皆さん を 旅 に お連れ します [VP end]

Oda, Yusuke et al., Syntax-based Simultaneous Translation through Prediction of Unseen Syntactic Constituents, Proc. of ACL-IJCNLP 2015.
Y.Kano, K.Sudo, S.Nakamura, “Simultaneous Neural Machine Translation with Constituent Label Prediction”, Proc. of WMT 2021.
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

23

Cascade Simultaneous S2S Translation System

I am a chef and a foodpolicy guy,
but I come from ...

Input English
speech

Output Japanese
speech

私はシェフで、食べ物政策の人…

Incremental

Incremental

Incremental

ASR
(ISR)

MT
(IMT)

TTS
(ITTS)

Intermediate
English transcript

I was a chef and a foodpolicy guy,
but I come from ...

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

私はシェフで、食べ物 Intermediate
Japanese MT result
政策の人…

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

24

Incremental Speech Recognition (ISR) [Novitasari, 2020]

S.Novitasari, A.Tjandra, T.Yanagita, S.Sakti, S.Nakamura, “Incremental Machine Speech Chain Towards Enabling Listening while
Speaking in Real-time”, Proceedings of INTERSPEECH 2020, Oct. 2020
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

25

Incremental Speech Synthesis (iTTS)[Novitasari 2021]

S.Novitasari, S.Sakti, S.Nakamura, “Dynamically Adaptive Machine Speech Chain Inference for TTS in Noisy Environment:
Listen and Speak Louder”, Proc. Interspeech 2021, 4124-4128, Aug. 30, 2021
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

26

Video

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

Problem in the current Simultaneous Speech Translation System
Source speech spectrogram (English)

Seriousdelay

Target speech spectrogram(Japanese)

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

27

28

Evaluation Overview
The cascade system was evaluated using TED Talks data in:– System-level latency• Ear-Voice Span (EVS)– Span between the start of the speaker’s speech input & interpreter’s speech output– Module-level quality• ISR: Word Error Rate (WER)• ISR+IMT: BLEU
• ITTS: L2-norm loss and subjective evaluation (AB preference test)

Three latency regimes: low, medium, high (following IWSLT)

R.Fukuda, et al, “SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM WITH TRANSFORMER-BASED INCREMENTAL ASR, MT,
AND TTS”, Proc. Oriental COCOSDA 2021
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

29

Overview of our corpus
Over 300 hours* = interpreted by interpreters from all 3 ranks (4h x 3 interpreters)
Others = interpreted by either an S- or A-rank interpreter
About half of the SIs have been transcribed
Direction

Source

2018

2019

2020

Experience

Rank

En → Ja

TED

67+12*

50

50

15 yearsS-rankJp → En

TEDx

12*

40

0

4 years

A-rank

CSJ

33

0

0

1 years

B-rank

JNPC

4

36.5

0

Total

128

126.5

50

Cum.

128

254.5

304.5

Detailed analysis: K.Doi, K.Sudoh, S.Nakamura, “Large-scale English-Japanese Simultanous Interpretation Corpus:
Construction and Analysis with sentence-aligned data, Proc. IWSLT 2021
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

30

Evaluation: Quality (ASR & MT for TED Talks)
ISR+IMT

ISR
Model
LSTM
Nonincremental Transformer
LSTM(low latency)

LSTM
Incremental(high latency)

Transformer(low latency)

Transformer(high latency)

BLEU- Subjective evaluation
4 (%) Adequacy Fluency

WER (%)
25.46
20.74
31.88
32.43

Gold transcript+MT

15.7

3.41

3.93

Non-incremental
ASR+MT

12.8

3.20

4.01

IMT

5.1

2.80

3.03

(low latency EVS:8.81s)

IMT

32.06

(medium latency
EVS:11.87s)

8.4

2.98

3.54

25.01

IMT

9.4

3.34

3.80

(high latency EVS:16.91s)

R.Fukuda, et al, “SIMULTANEOUS SPEECH-TO-SPEECH TRANSLATION SYSTEM WITH TRANSFORMER-BASED INCREMENTAL ASR, MT,
AND TTS”, Proc. Oriental COCOSDA 2021
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

31

Summary
Remarkable progress– Statistical Models– Deep Neural Network– Progress in Speech Translation

Automatic Simultaneous Speech Translation (Interpretation)– Data Collection– Automatic Simultaneous Speech Translation for distant language pairs

Further Research– Higher Quality with Lower Latency– Evaluation of Simultaneous Speech Translation– Context/ Situation dependency– Semantics, Discourse Analysis– Para-linguistics/ Multi-modal

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/

32

Thank you for listening

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 12 2021

©Satoshi Nakamura, AHC Lab, NAIST, Japan. AIST 2021 @IGDTUW India

http://www.naist.jp/